Nov 03 20:47:12.998382 ip-10-0-129-192 systemd[1]: Starting Kubernetes Kubelet... Nov 03 20:47:13.464894 ip-10-0-129-192 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:47:13.464894 ip-10-0-129-192 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 20:47:13.464894 ip-10-0-129-192 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:47:13.464894 ip-10-0-129-192 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:47:13.469784 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469699 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:47:13.469784 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469782 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:47:13.469784 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469786 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:47:13.469784 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469789 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:47:13.469950 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469792 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:47:13.469950 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469796 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:47:13.469950 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469798 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:47:13.469950 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469801 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:47:13.469950 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469803 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:47:13.469950 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469806 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:47:13.469950 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469809 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:47:13.469950 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469811 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:47:13.469950 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469814 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:47:13.469950 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469817 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:47:13.469950 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469819 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:47:13.469950 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469822 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:47:13.469950 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469824 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:47:13.469950 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469827 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:47:13.469950 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469830 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:47:13.469950 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469833 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:47:13.469950 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469841 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:47:13.469950 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469845 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:47:13.469950 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469847 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:47:13.470579 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469850 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:47:13.470579 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469852 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:47:13.470579 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469855 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:47:13.470579 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469858 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:47:13.470579 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469860 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:47:13.470579 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469863 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:47:13.470579 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469865 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:47:13.470579 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469868 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:47:13.470579 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469872 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:47:13.470579 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469874 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:47:13.470579 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469877 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:47:13.470579 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469880 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:47:13.470579 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469884 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:47:13.470579 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469889 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:47:13.470579 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469893 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:47:13.470579 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469896 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:47:13.470579 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469899 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:47:13.470579 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469903 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:47:13.470579 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469905 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:47:13.470579 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469909 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:47:13.471417 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469912 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:47:13.471417 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469914 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:47:13.471417 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469917 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:47:13.471417 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469920 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:47:13.471417 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469922 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:47:13.471417 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469925 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:47:13.471417 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469927 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:47:13.471417 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469930 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:47:13.471417 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469932 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:47:13.471417 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469935 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:47:13.471417 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469938 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:47:13.471417 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469940 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:47:13.471417 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469942 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:47:13.471417 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469945 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:47:13.471417 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469948 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:47:13.471417 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469950 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:47:13.471417 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469953 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:47:13.471417 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469955 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:47:13.471417 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469958 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:47:13.472031 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469961 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:47:13.472031 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469964 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:47:13.472031 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469966 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:47:13.472031 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469969 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:47:13.472031 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469972 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:47:13.472031 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469975 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:47:13.472031 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469978 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:47:13.472031 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469981 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:47:13.472031 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469983 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:47:13.472031 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469986 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:47:13.472031 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469989 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:47:13.472031 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469992 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:47:13.472031 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469994 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:47:13.472031 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469997 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:47:13.472031 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.469999 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:47:13.472031 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470004 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:47:13.472031 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470008 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:47:13.472031 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470011 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:47:13.472031 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470014 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:47:13.472696 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470016 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:47:13.472696 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470019 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:47:13.472696 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470022 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:47:13.472696 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470024 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:47:13.472696 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470027 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:47:13.472696 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470029 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:47:13.472696 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470032 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:47:13.472696 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470035 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:47:13.472696 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470038 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:47:13.472696 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470040 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:47:13.472696 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470043 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:47:13.472696 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470045 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:47:13.472696 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470048 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:47:13.472696 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470051 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:47:13.472696 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470053 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:47:13.472696 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470056 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:47:13.472696 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470058 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:47:13.472696 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470062 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:47:13.472696 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470064 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:47:13.472696 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470067 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:47:13.473394 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470085 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:47:13.473394 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470088 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:47:13.473394 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470091 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:47:13.473394 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470093 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:47:13.473394 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470096 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:47:13.473394 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470898 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:47:13.473394 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470904 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:47:13.473394 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470906 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:47:13.473394 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470910 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:47:13.473394 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470912 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:47:13.473394 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470915 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:47:13.473394 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470917 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:47:13.473394 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470921 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:47:13.473394 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470924 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:47:13.473394 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470927 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:47:13.473394 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470929 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:47:13.473394 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470932 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:47:13.473394 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470934 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:47:13.473394 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470937 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:47:13.473394 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470939 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:47:13.474082 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470942 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:47:13.474082 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470944 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:47:13.474082 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470947 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:47:13.474082 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470949 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:47:13.474082 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470953 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:47:13.474082 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470955 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:47:13.474082 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470958 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:47:13.474082 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470961 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:47:13.474082 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470963 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:47:13.474082 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470965 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:47:13.474082 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470968 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:47:13.474082 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470970 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:47:13.474082 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470973 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:47:13.474082 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470976 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:47:13.474082 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470979 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:47:13.474082 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470981 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:47:13.474082 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470984 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:47:13.474082 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470987 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:47:13.474082 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470990 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:47:13.474768 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470993 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:47:13.474768 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470995 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:47:13.474768 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.470998 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:47:13.474768 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471000 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:47:13.474768 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471002 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:47:13.474768 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471005 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:47:13.474768 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471008 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:47:13.474768 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471010 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:47:13.474768 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471013 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:47:13.474768 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471015 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:47:13.474768 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471017 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:47:13.474768 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471020 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:47:13.474768 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471022 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:47:13.474768 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471025 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:47:13.474768 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471028 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:47:13.474768 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471030 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:47:13.474768 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471033 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:47:13.474768 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471036 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:47:13.474768 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471038 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:47:13.474768 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471041 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:47:13.475463 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471044 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:47:13.475463 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471047 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:47:13.475463 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471049 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:47:13.475463 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471052 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:47:13.475463 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471054 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:47:13.475463 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471057 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:47:13.475463 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471060 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:47:13.475463 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471062 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:47:13.475463 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471065 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:47:13.475463 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471067 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:47:13.475463 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471070 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:47:13.475463 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471088 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:47:13.475463 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471091 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:47:13.475463 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471094 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:47:13.475463 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471096 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:47:13.475463 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471099 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:47:13.475463 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471102 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:47:13.475463 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471104 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:47:13.475463 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471107 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:47:13.475463 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471110 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:47:13.476145 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471112 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:47:13.476145 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471115 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:47:13.476145 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471117 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:47:13.476145 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471119 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:47:13.476145 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471123 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:47:13.476145 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471127 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:47:13.476145 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471129 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:47:13.476145 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471132 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:47:13.476145 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471136 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:47:13.476145 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471139 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:47:13.476145 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471142 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:47:13.476145 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471145 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:47:13.476145 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471147 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:47:13.476145 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471150 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:47:13.476145 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471152 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:47:13.476145 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471155 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:47:13.476145 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471157 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:47:13.476145 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471160 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:47:13.476145 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471163 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:47:13.476786 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471165 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:47:13.476786 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471168 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:47:13.476786 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471171 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:47:13.476786 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471173 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:47:13.476786 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471176 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:47:13.476786 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471179 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:47:13.476786 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471182 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:47:13.476786 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471184 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:47:13.476786 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471187 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:47:13.476786 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471191 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:47:13.476786 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471196 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:47:13.476786 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471198 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:47:13.476786 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.471201 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:47:13.476786 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473016 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:47:13.476786 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473024 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:47:13.476786 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473027 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:47:13.476786 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473030 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:47:13.476786 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473033 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:47:13.476786 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473035 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:47:13.477435 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473038 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:47:13.477435 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473040 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:47:13.477435 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473043 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:47:13.477435 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473046 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:47:13.477435 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473048 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:47:13.477435 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473051 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:47:13.477435 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473053 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:47:13.477435 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473056 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:47:13.477435 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473058 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:47:13.477435 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473061 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:47:13.477435 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473063 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:47:13.477435 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473067 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:47:13.477435 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473070 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:47:13.477435 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473083 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:47:13.477435 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473085 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:47:13.477435 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473088 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:47:13.477435 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473091 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:47:13.477435 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473094 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:47:13.477435 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473098 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:47:13.477435 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473100 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:47:13.478108 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473103 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:47:13.478108 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473106 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:47:13.478108 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473109 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:47:13.478108 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473111 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:47:13.478108 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473114 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:47:13.478108 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473116 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:47:13.478108 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473119 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:47:13.478108 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473121 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:47:13.478108 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473124 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:47:13.478108 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473126 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:47:13.478108 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473129 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:47:13.478108 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473131 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:47:13.478108 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473134 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:47:13.478108 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473136 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:47:13.478108 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473139 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:47:13.478108 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473141 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:47:13.478108 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473144 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:47:13.478108 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473148 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:47:13.478108 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473152 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:47:13.478108 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473155 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:47:13.478781 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473157 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:47:13.478781 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473160 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:47:13.478781 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473164 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:47:13.478781 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473166 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:47:13.478781 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473169 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:47:13.478781 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473172 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:47:13.478781 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473174 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:47:13.478781 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473177 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:47:13.478781 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473180 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:47:13.478781 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473183 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:47:13.478781 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473185 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:47:13.478781 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473189 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:47:13.478781 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473191 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:47:13.478781 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473194 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:47:13.478781 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473196 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:47:13.478781 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473199 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:47:13.478781 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473201 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:47:13.478781 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473204 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:47:13.478781 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473206 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:47:13.478781 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473209 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:47:13.479461 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473211 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:47:13.479461 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473214 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:47:13.479461 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473217 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:47:13.479461 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473220 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:47:13.479461 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473223 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:47:13.479461 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473225 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:47:13.479461 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473228 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:47:13.479461 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473230 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:47:13.479461 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473233 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:47:13.479461 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473236 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:47:13.479461 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473238 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:47:13.479461 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473241 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:47:13.479461 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473243 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:47:13.479461 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473246 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:47:13.479461 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473248 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:47:13.479461 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473252 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:47:13.479461 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473256 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:47:13.479461 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473259 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:47:13.479461 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473262 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:47:13.480135 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473265 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:47:13.480135 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473267 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:47:13.480135 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473270 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:47:13.480135 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473272 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:47:13.480135 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473275 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:47:13.480135 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473278 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:47:13.480135 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473280 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:47:13.480135 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473283 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:47:13.480135 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473285 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:47:13.480135 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473288 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:47:13.480135 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473290 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:47:13.480135 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473293 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:47:13.480135 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473295 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:47:13.480135 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473298 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:47:13.480135 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473300 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:47:13.480135 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473303 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:47:13.480135 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473305 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:47:13.480135 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473308 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:47:13.480741 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473310 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:47:13.480741 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473313 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:47:13.480741 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.473316 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:47:13.480741 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475165 2575 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 20:47:13.480741 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475181 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 20:47:13.480741 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475188 2575 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 20:47:13.480741 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475193 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 20:47:13.480741 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475197 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 20:47:13.480741 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475200 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 20:47:13.480741 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475205 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 20:47:13.480741 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475209 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 20:47:13.480741 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475212 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 20:47:13.480741 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475216 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 20:47:13.480741 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475219 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 20:47:13.480741 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475223 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 20:47:13.480741 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475226 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 20:47:13.480741 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475229 2575 flags.go:64] FLAG: --cgroup-root="" Nov 03 20:47:13.480741 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475232 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 20:47:13.480741 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475235 2575 flags.go:64] FLAG: --client-ca-file="" Nov 03 20:47:13.480741 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475238 2575 flags.go:64] FLAG: --cloud-provider="external" Nov 03 20:47:13.480741 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475243 2575 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 20:47:13.480741 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475248 2575 flags.go:64] FLAG: --cluster-domain="" Nov 03 20:47:13.480741 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475251 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 20:47:13.481521 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475254 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 20:47:13.481521 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475257 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 20:47:13.481521 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475261 2575 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 20:47:13.481521 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475265 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 20:47:13.481521 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475269 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 20:47:13.481521 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475272 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 20:47:13.481521 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475275 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 20:47:13.481521 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475278 2575 flags.go:64] FLAG: --contention-profiling="false" Nov 03 20:47:13.481521 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475282 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 20:47:13.481521 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475285 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 20:47:13.481521 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475288 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 20:47:13.481521 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475291 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 20:47:13.481521 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475295 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 20:47:13.481521 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475298 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 20:47:13.481521 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475301 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 20:47:13.481521 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475304 2575 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 20:47:13.481521 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475307 2575 flags.go:64] FLAG: --enable-server="true" Nov 03 20:47:13.481521 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475310 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 20:47:13.481521 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475315 2575 flags.go:64] FLAG: --event-burst="100" Nov 03 20:47:13.481521 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475318 2575 flags.go:64] FLAG: --event-qps="50" Nov 03 20:47:13.481521 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475321 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 20:47:13.481521 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475325 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 20:47:13.481521 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475329 2575 flags.go:64] FLAG: --eviction-hard="" Nov 03 20:47:13.481521 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475333 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 20:47:13.482399 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475336 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 20:47:13.482399 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475340 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 20:47:13.482399 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475343 2575 flags.go:64] FLAG: --eviction-soft="" Nov 03 20:47:13.482399 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475346 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 20:47:13.482399 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475349 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 20:47:13.482399 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475352 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 20:47:13.482399 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475356 2575 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 20:47:13.482399 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475359 2575 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 20:47:13.482399 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475362 2575 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 20:47:13.482399 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475365 2575 flags.go:64] FLAG: --feature-gates="" Nov 03 20:47:13.482399 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475369 2575 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 20:47:13.482399 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475372 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 20:47:13.482399 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475376 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 20:47:13.482399 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475379 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 20:47:13.482399 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475382 2575 flags.go:64] FLAG: --healthz-port="10248" Nov 03 20:47:13.482399 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475385 2575 flags.go:64] FLAG: --help="false" Nov 03 20:47:13.482399 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475388 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:47:13.482399 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475392 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 20:47:13.482399 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475395 2575 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 20:47:13.482399 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475398 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 20:47:13.482399 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475402 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 20:47:13.482399 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475405 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 20:47:13.482399 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475408 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 20:47:13.483183 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475411 2575 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 20:47:13.483183 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475414 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 20:47:13.483183 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475417 2575 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 20:47:13.483183 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475423 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 20:47:13.483183 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475427 2575 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 20:47:13.483183 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475430 2575 flags.go:64] FLAG: --kube-reserved="" Nov 03 20:47:13.483183 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475435 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 20:47:13.483183 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475438 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 20:47:13.483183 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475441 2575 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 20:47:13.483183 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475444 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 20:47:13.483183 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475447 2575 flags.go:64] FLAG: --lock-file="" Nov 03 20:47:13.483183 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475450 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 20:47:13.483183 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475453 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 20:47:13.483183 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475456 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 20:47:13.483183 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475462 2575 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 20:47:13.483183 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475465 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 20:47:13.483183 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475469 2575 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 20:47:13.483183 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475473 2575 flags.go:64] FLAG: --logging-format="text" Nov 03 20:47:13.483183 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475476 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 20:47:13.483183 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475480 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 20:47:13.483183 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475483 2575 flags.go:64] FLAG: --manifest-url="" Nov 03 20:47:13.483183 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475485 2575 flags.go:64] FLAG: --manifest-url-header="" Nov 03 20:47:13.483183 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475491 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 20:47:13.483183 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475494 2575 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 20:47:13.483183 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475498 2575 flags.go:64] FLAG: --max-pods="110" Nov 03 20:47:13.484027 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475501 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 20:47:13.484027 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475504 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 20:47:13.484027 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475508 2575 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 20:47:13.484027 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475510 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 20:47:13.484027 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475513 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 20:47:13.484027 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475516 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 20:47:13.484027 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475519 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 20:47:13.484027 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475526 2575 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 20:47:13.484027 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475530 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 20:47:13.484027 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475533 2575 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 20:47:13.484027 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475537 2575 flags.go:64] FLAG: --pod-cidr="" Nov 03 20:47:13.484027 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475540 2575 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 20:47:13.484027 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475543 2575 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 20:47:13.484027 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475547 2575 flags.go:64] FLAG: --pods-per-core="0" Nov 03 20:47:13.484027 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475550 2575 flags.go:64] FLAG: --port="10250" Nov 03 20:47:13.484027 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475554 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 20:47:13.484027 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475557 2575 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0eec255b93f603248" Nov 03 20:47:13.484027 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475560 2575 flags.go:64] FLAG: --qos-reserved="" Nov 03 20:47:13.484027 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475563 2575 flags.go:64] FLAG: --read-only-port="10255" Nov 03 20:47:13.484027 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475566 2575 flags.go:64] FLAG: --register-node="true" Nov 03 20:47:13.484027 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475569 2575 flags.go:64] FLAG: --register-with-taints="" Nov 03 20:47:13.484027 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475573 2575 flags.go:64] FLAG: --registry-burst="10" Nov 03 20:47:13.484027 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475576 2575 flags.go:64] FLAG: --registry-qps="5" Nov 03 20:47:13.484027 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475579 2575 flags.go:64] FLAG: --reserved-cpus="" Nov 03 20:47:13.484027 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475582 2575 flags.go:64] FLAG: --reserved-memory="" Nov 03 20:47:13.484027 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475586 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 20:47:13.484930 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475589 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 20:47:13.484930 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475593 2575 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 20:47:13.484930 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475596 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 20:47:13.484930 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475599 2575 flags.go:64] FLAG: --runonce="false" Nov 03 20:47:13.484930 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475602 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 20:47:13.484930 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475605 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 20:47:13.484930 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475608 2575 flags.go:64] FLAG: --seccomp-default="false" Nov 03 20:47:13.484930 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475611 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 20:47:13.484930 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475614 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 20:47:13.484930 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475617 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 20:47:13.484930 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475620 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 20:47:13.484930 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475623 2575 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 20:47:13.484930 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475626 2575 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 20:47:13.484930 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475629 2575 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 20:47:13.484930 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475632 2575 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 20:47:13.484930 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475635 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 20:47:13.484930 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475639 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 20:47:13.484930 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475642 2575 flags.go:64] FLAG: --system-cgroups="" Nov 03 20:47:13.484930 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475645 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 20:47:13.484930 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475652 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 20:47:13.484930 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475655 2575 flags.go:64] FLAG: --tls-cert-file="" Nov 03 20:47:13.484930 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475658 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 20:47:13.484930 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475662 2575 flags.go:64] FLAG: --tls-min-version="" Nov 03 20:47:13.484930 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475665 2575 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 20:47:13.484930 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475669 2575 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 20:47:13.485849 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475671 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 20:47:13.485849 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475675 2575 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 20:47:13.485849 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475678 2575 flags.go:64] FLAG: --v="2" Nov 03 20:47:13.485849 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475687 2575 flags.go:64] FLAG: --version="false" Nov 03 20:47:13.485849 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475691 2575 flags.go:64] FLAG: --vmodule="" Nov 03 20:47:13.485849 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475695 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 20:47:13.485849 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.475699 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 20:47:13.485849 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475842 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:47:13.485849 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475846 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:47:13.485849 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475849 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:47:13.485849 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475852 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:47:13.485849 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475856 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:47:13.485849 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475858 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:47:13.485849 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475862 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:47:13.485849 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475866 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:47:13.485849 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475870 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:47:13.485849 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475873 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:47:13.485849 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475877 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:47:13.485849 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475879 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:47:13.485849 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475882 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:47:13.485849 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475885 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:47:13.486603 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475888 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:47:13.486603 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475890 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:47:13.486603 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475894 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:47:13.486603 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475898 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:47:13.486603 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475900 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:47:13.486603 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475904 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:47:13.486603 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475907 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:47:13.486603 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475910 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:47:13.486603 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475912 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:47:13.486603 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475915 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:47:13.486603 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475918 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:47:13.486603 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475922 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:47:13.486603 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475926 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:47:13.486603 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475929 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:47:13.486603 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475932 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:47:13.486603 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475935 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:47:13.486603 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475937 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:47:13.486603 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475940 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:47:13.486603 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475943 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:47:13.487279 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475946 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:47:13.487279 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475949 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:47:13.487279 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475951 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:47:13.487279 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475954 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:47:13.487279 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475957 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:47:13.487279 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475959 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:47:13.487279 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475962 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:47:13.487279 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475965 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:47:13.487279 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475967 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:47:13.487279 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475970 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:47:13.487279 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475972 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:47:13.487279 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475975 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:47:13.487279 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475977 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:47:13.487279 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475980 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:47:13.487279 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475982 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:47:13.487279 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475986 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:47:13.487279 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475989 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:47:13.487279 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475992 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:47:13.487279 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475996 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:47:13.487923 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.475999 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:47:13.487923 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476001 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:47:13.487923 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476004 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:47:13.487923 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476007 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:47:13.487923 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476010 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:47:13.487923 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476012 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:47:13.487923 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476015 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:47:13.487923 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476017 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:47:13.487923 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476020 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:47:13.487923 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476023 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:47:13.487923 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476025 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:47:13.487923 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476028 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:47:13.487923 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476030 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:47:13.487923 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476034 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:47:13.487923 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476036 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:47:13.487923 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476039 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:47:13.487923 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476041 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:47:13.487923 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476044 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:47:13.487923 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476047 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:47:13.487923 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476049 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:47:13.488602 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476052 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:47:13.488602 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476055 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:47:13.488602 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476057 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:47:13.488602 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476060 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:47:13.488602 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476062 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:47:13.488602 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476065 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:47:13.488602 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476067 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:47:13.488602 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476070 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:47:13.488602 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476084 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:47:13.488602 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476087 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:47:13.488602 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476091 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:47:13.488602 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476095 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:47:13.488602 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476097 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:47:13.488602 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476100 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:47:13.488602 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476102 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:47:13.488602 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476105 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:47:13.488602 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476108 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:47:13.488602 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476110 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:47:13.488602 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476113 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:47:13.488602 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476115 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:47:13.489286 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476118 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:47:13.489286 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476120 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:47:13.489286 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476123 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:47:13.489286 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476125 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:47:13.489286 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476128 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:47:13.489286 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476131 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:47:13.489286 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476134 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:47:13.489286 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476137 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:47:13.489286 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476140 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:47:13.489286 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476142 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:47:13.489286 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476145 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:47:13.489286 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476147 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:47:13.489286 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476150 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:47:13.489286 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476153 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:47:13.489286 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.476180 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:47:13.489286 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476216 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:47:13.489837 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476221 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:47:13.489837 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476224 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:47:13.489837 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476227 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:47:13.489837 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476230 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:47:13.489837 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476233 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:47:13.489837 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476237 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:47:13.489837 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476240 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:47:13.489837 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476245 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:47:13.489837 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476248 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:47:13.489837 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476251 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:47:13.489837 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476253 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:47:13.489837 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476256 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:47:13.489837 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476258 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:47:13.489837 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476261 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:47:13.489837 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476264 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:47:13.489837 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476266 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:47:13.489837 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476269 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:47:13.489837 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476272 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:47:13.489837 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476274 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:47:13.490537 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476278 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:47:13.490537 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476280 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:47:13.490537 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476283 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:47:13.490537 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476286 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:47:13.490537 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476289 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:47:13.490537 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476292 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:47:13.490537 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476294 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:47:13.490537 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476298 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:47:13.490537 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476301 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:47:13.490537 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476304 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:47:13.490537 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476308 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:47:13.490537 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476311 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:47:13.490537 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476314 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:47:13.490537 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476316 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:47:13.490537 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476319 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:47:13.490537 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476322 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:47:13.490537 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476325 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:47:13.490537 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476327 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:47:13.490537 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476330 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:47:13.491189 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476332 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:47:13.491189 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476336 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:47:13.491189 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476339 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:47:13.491189 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476342 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:47:13.491189 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476344 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:47:13.491189 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476346 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:47:13.491189 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476349 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:47:13.491189 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476353 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:47:13.491189 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476356 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:47:13.491189 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476359 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:47:13.491189 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476362 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:47:13.491189 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476365 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:47:13.491189 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476368 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:47:13.491189 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476371 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:47:13.491189 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476374 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:47:13.491189 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476377 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:47:13.491189 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476380 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:47:13.491189 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476383 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:47:13.491189 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476386 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:47:13.491818 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476389 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:47:13.491818 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476392 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:47:13.491818 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476394 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:47:13.491818 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476397 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:47:13.491818 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476400 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:47:13.491818 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476402 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:47:13.491818 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476405 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:47:13.491818 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476407 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:47:13.491818 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476410 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:47:13.491818 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476412 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:47:13.491818 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476416 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:47:13.491818 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476418 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:47:13.491818 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476421 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:47:13.491818 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476424 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:47:13.491818 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476428 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:47:13.491818 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476431 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:47:13.491818 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476433 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:47:13.491818 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476436 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:47:13.491818 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476438 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:47:13.492455 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476441 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:47:13.492455 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476444 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:47:13.492455 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476446 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:47:13.492455 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476449 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:47:13.492455 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476451 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:47:13.492455 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476454 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:47:13.492455 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476456 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:47:13.492455 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476459 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:47:13.492455 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476462 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:47:13.492455 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476464 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:47:13.492455 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476466 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:47:13.492455 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476469 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:47:13.492455 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476472 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:47:13.492455 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476475 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:47:13.492455 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476478 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:47:13.492455 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476480 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:47:13.492455 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476484 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:47:13.492455 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476486 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:47:13.492455 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476489 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:47:13.492455 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476497 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:47:13.493139 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476502 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:47:13.493139 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476506 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:47:13.493139 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476509 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:47:13.493139 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476511 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:47:13.493139 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476514 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:47:13.493139 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476516 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:47:13.493139 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476519 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:47:13.493139 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476523 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:47:13.493139 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.476526 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:47:13.493139 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.484395 2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 20:47:13.493139 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.484411 2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 20:47:13.493139 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.484428 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 20:47:13.493139 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.484432 2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 20:47:13.493139 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484501 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:47:13.493139 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484507 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:47:13.493139 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484511 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:47:13.493139 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484515 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:47:13.493139 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484518 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:47:13.493139 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484520 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:47:13.493139 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484523 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:47:13.493801 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484526 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:47:13.493801 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484542 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:47:13.493801 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484547 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:47:13.493801 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484550 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:47:13.493801 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484553 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:47:13.493801 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484557 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:47:13.493801 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484560 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:47:13.493801 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484563 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:47:13.493801 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484566 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:47:13.493801 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484569 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:47:13.493801 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484572 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:47:13.493801 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484574 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:47:13.493801 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484577 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:47:13.493801 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484580 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:47:13.493801 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484582 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:47:13.493801 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484585 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:47:13.493801 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484589 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:47:13.493801 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484592 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:47:13.493801 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484595 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:47:13.494451 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484598 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:47:13.494451 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484601 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:47:13.494451 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484604 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:47:13.494451 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484608 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:47:13.494451 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484612 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:47:13.494451 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484614 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:47:13.494451 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484617 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:47:13.494451 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484620 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:47:13.494451 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484623 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:47:13.494451 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484626 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:47:13.494451 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484629 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:47:13.494451 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484631 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:47:13.494451 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484633 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:47:13.494451 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484636 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:47:13.494451 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484639 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:47:13.494451 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484641 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:47:13.494451 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484644 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:47:13.494451 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484646 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:47:13.494451 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484649 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:47:13.495183 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484652 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:47:13.495183 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484655 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:47:13.495183 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484658 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:47:13.495183 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484660 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:47:13.495183 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484663 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:47:13.495183 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484665 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:47:13.495183 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484668 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:47:13.495183 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484670 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:47:13.495183 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484672 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:47:13.495183 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484675 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:47:13.495183 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484678 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:47:13.495183 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484680 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:47:13.495183 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484683 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:47:13.495183 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484686 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:47:13.495183 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484689 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:47:13.495183 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484691 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:47:13.495183 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484694 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:47:13.495183 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484697 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:47:13.495183 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484699 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:47:13.495183 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484702 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:47:13.495850 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484704 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:47:13.495850 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484707 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:47:13.495850 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484709 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:47:13.495850 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484712 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:47:13.495850 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484714 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:47:13.495850 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484717 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:47:13.495850 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484719 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:47:13.495850 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484722 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:47:13.495850 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484724 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:47:13.495850 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484727 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:47:13.495850 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484729 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:47:13.495850 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484732 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:47:13.495850 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484734 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:47:13.495850 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484737 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:47:13.495850 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484740 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:47:13.495850 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484742 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:47:13.495850 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484745 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:47:13.495850 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484747 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:47:13.495850 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484750 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:47:13.496562 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484752 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:47:13.496562 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484755 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:47:13.496562 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484757 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:47:13.496562 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484760 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:47:13.496562 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484762 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:47:13.496562 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484764 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:47:13.496562 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484767 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:47:13.496562 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484770 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:47:13.496562 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484772 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:47:13.496562 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484775 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:47:13.496562 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484778 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:47:13.496562 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484780 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:47:13.496562 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484783 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:47:13.496562 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484785 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:47:13.496562 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484788 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:47:13.496562 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484791 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:47:13.496562 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484793 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:47:13.496562 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484796 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:47:13.496562 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484799 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:47:13.497202 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484801 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:47:13.497202 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484804 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:47:13.497202 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484806 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:47:13.497202 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.484834 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:47:13.497202 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484979 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:47:13.497202 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484984 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:47:13.497202 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484987 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:47:13.497202 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484990 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:47:13.497202 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484994 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:47:13.497202 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.484997 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:47:13.497202 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485000 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:47:13.497202 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485002 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:47:13.497202 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485005 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:47:13.497202 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485007 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:47:13.497202 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485010 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:47:13.497202 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485012 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:47:13.497731 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485015 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:47:13.497731 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485018 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:47:13.497731 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485020 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:47:13.497731 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485023 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:47:13.497731 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485026 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:47:13.497731 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485028 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:47:13.497731 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485031 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:47:13.497731 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485034 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:47:13.497731 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485036 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:47:13.497731 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485039 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:47:13.497731 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485042 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:47:13.497731 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485044 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:47:13.497731 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485047 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:47:13.497731 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485049 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:47:13.497731 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485052 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:47:13.497731 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485054 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:47:13.497731 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485057 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:47:13.497731 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485059 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:47:13.497731 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485062 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:47:13.497731 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485064 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:47:13.498405 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485067 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:47:13.498405 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485069 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:47:13.498405 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485086 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:47:13.498405 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485090 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:47:13.498405 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485094 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:47:13.498405 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485098 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:47:13.498405 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485102 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:47:13.498405 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485106 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:47:13.498405 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485109 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:47:13.498405 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485112 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:47:13.498405 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485115 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:47:13.498405 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485117 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:47:13.498405 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485120 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:47:13.498405 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485122 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:47:13.498405 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485125 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:47:13.498405 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485128 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:47:13.498405 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485130 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:47:13.498405 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485133 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:47:13.498405 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485136 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:47:13.498405 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485139 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:47:13.499056 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485142 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:47:13.499056 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485144 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:47:13.499056 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485147 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:47:13.499056 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485150 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:47:13.499056 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485152 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:47:13.499056 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485155 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:47:13.499056 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485158 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:47:13.499056 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485160 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:47:13.499056 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485163 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:47:13.499056 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485165 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:47:13.499056 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485168 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:47:13.499056 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485170 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:47:13.499056 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485173 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:47:13.499056 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485175 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:47:13.499056 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485178 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:47:13.499056 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485180 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:47:13.499056 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485183 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:47:13.499056 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485185 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:47:13.499056 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485189 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:47:13.499816 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485192 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:47:13.499816 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485194 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:47:13.499816 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485197 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:47:13.499816 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485200 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:47:13.499816 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485202 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:47:13.499816 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485206 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:47:13.499816 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485210 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:47:13.499816 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485214 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:47:13.499816 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485217 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:47:13.499816 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485220 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:47:13.499816 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485222 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:47:13.499816 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485225 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:47:13.499816 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485227 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:47:13.499816 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485230 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:47:13.499816 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485232 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:47:13.499816 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485235 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:47:13.499816 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485237 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:47:13.499816 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485240 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:47:13.499816 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485242 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:47:13.500464 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485245 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:47:13.500464 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485248 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:47:13.500464 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485250 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:47:13.500464 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485253 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:47:13.500464 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485255 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:47:13.500464 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485258 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:47:13.500464 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485260 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:47:13.500464 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485262 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:47:13.500464 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485265 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:47:13.500464 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485268 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:47:13.500464 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485271 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:47:13.500464 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485273 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:47:13.500464 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485276 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:47:13.500464 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485279 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:47:13.500464 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485282 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:47:13.500464 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485286 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:47:13.500990 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.485310 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:47:13.500990 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485351 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:47:13.500990 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485355 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:47:13.500990 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485359 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:47:13.500990 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485362 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:47:13.500990 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485365 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:47:13.500990 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485367 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:47:13.500990 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485370 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:47:13.500990 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485373 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:47:13.500990 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485375 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:47:13.500990 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485378 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:47:13.500990 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485380 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:47:13.500990 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485383 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:47:13.500990 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485385 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:47:13.500990 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485388 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:47:13.500990 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485391 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:47:13.501532 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485393 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:47:13.501532 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485396 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:47:13.501532 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485398 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:47:13.501532 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485401 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:47:13.501532 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485403 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:47:13.501532 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485406 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:47:13.501532 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485409 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:47:13.501532 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485411 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:47:13.501532 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485414 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:47:13.501532 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485416 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:47:13.501532 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485420 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:47:13.501532 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485424 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:47:13.501532 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485427 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:47:13.501532 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485430 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:47:13.501532 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485433 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:47:13.501532 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485436 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:47:13.501532 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485440 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:47:13.501532 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485442 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:47:13.501532 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485445 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:47:13.502169 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485448 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:47:13.502169 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485450 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:47:13.502169 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485453 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:47:13.502169 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485456 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:47:13.502169 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485458 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:47:13.502169 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485461 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:47:13.502169 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485463 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:47:13.502169 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485466 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:47:13.502169 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485468 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:47:13.502169 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485471 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:47:13.502169 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485473 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:47:13.502169 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485476 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:47:13.502169 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485478 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:47:13.502169 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485481 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:47:13.502169 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485484 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:47:13.502169 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485486 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:47:13.502169 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485489 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:47:13.502169 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485491 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:47:13.502169 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485494 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:47:13.502799 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485496 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:47:13.502799 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485499 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:47:13.502799 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485501 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:47:13.502799 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485504 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:47:13.502799 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485507 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:47:13.502799 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485509 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:47:13.502799 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485512 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:47:13.502799 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485515 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:47:13.502799 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485517 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:47:13.502799 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485520 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:47:13.502799 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485522 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:47:13.502799 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485525 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:47:13.502799 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485527 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:47:13.502799 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485530 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:47:13.502799 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485533 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:47:13.502799 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485535 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:47:13.502799 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485541 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:47:13.502799 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485543 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:47:13.502799 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485546 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:47:13.502799 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485548 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:47:13.503554 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485551 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:47:13.503554 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485554 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:47:13.503554 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485556 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:47:13.503554 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485560 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:47:13.503554 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485563 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:47:13.503554 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485565 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:47:13.503554 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485568 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:47:13.503554 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485571 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:47:13.503554 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485574 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:47:13.503554 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485576 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:47:13.503554 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485579 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:47:13.503554 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485581 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:47:13.503554 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485584 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:47:13.503554 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485587 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:47:13.503554 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485589 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:47:13.503554 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485592 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:47:13.503554 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485595 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:47:13.503554 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485597 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:47:13.503554 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485600 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:47:13.503554 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485602 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:47:13.504282 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485605 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:47:13.504282 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485607 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:47:13.504282 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485611 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:47:13.504282 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485613 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:47:13.504282 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485616 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:47:13.504282 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485619 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:47:13.504282 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485621 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:47:13.504282 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485624 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:47:13.504282 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485628 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:47:13.504282 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485632 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:47:13.504282 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485636 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:47:13.504282 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485639 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:47:13.504282 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:13.485641 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:47:13.504282 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.485698 2575 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 20:47:13.504282 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.487945 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 20:47:13.504282 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.488933 2575 server.go:1014] "Starting client certificate rotation" Nov 03 20:47:13.504282 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.489039 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 20:47:13.504282 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.489095 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 20:47:13.504862 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.494529 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:47:13.504862 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.495648 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:47:13.508164 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.508144 2575 log.go:25] "Validated CRI v1 runtime API" Nov 03 20:47:13.512129 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.512115 2575 log.go:25] "Validated CRI v1 image API" Nov 03 20:47:13.512990 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.512962 2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 20:47:13.519941 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.519910 2575 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 99e37395-ebd1-47ab-b414-5eda35e76ff4:/dev/nvme0n1p3 a0ca9470-af9b-4137-8afa-e76e68339521:/dev/nvme0n1p4] Nov 03 20:47:13.519941 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.519935 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 20:47:13.525953 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.525843 2575 manager.go:217] Machine: {Timestamp:2026-11-03 20:47:13.524301298 +0000 UTC m=+0.386176717 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101651 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e399285b3e2332cde22c26326417d SystemUUID:ec2e3992-85b3-e233-2cde-22c26326417d BootID:f8013594-09f1-4ea0-9b43-075a377e4ef2 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:f7:5e:b1:09 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:f7:5e:b1:09 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:9c:27:32:2a:d2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 20:47:13.525953 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.525950 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 20:47:13.526119 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.526103 2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 20:47:13.528830 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.528802 2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 20:47:13.528979 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.528833 2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-192.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 20:47:13.529020 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.528981 2575 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 20:47:13.529020 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.528989 2575 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 20:47:13.529020 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.529019 2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:47:13.529111 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.529042 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:47:13.530068 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.530056 2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 20:47:13.533581 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.533568 2575 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 20:47:13.533749 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.533730 2575 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 20:47:13.534547 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.534535 2575 kubelet.go:493] "Attempting to sync node with API server" Nov 03 20:47:13.534608 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.534549 2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 20:47:13.534608 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.534561 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 20:47:13.534608 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.534572 2575 kubelet.go:405] "Adding apiserver pod source" Nov 03 20:47:13.534608 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.534580 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 20:47:13.534997 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.534974 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:47:13.537362 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.537345 2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 20:47:13.543992 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.543978 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:47:13.545019 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.545003 2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 20:47:13.545069 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.545052 2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 20:47:13.545320 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.545304 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 20:47:13.545371 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.545335 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 20:47:13.545371 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.545347 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 20:47:13.545371 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.545355 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 20:47:13.545371 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.545361 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 20:47:13.545371 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.545367 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 20:47:13.546029 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.546018 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 20:47:13.546029 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.546028 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 20:47:13.546118 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.546045 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 20:47:13.546118 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.546056 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 20:47:13.546118 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.546081 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 20:47:13.546118 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.546090 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 20:47:13.546798 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.546786 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 20:47:13.546840 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.546801 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 20:47:13.551148 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.551123 2575 server.go:1263] "Started kubelet" Nov 03 20:47:13.555499 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.555430 2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 20:47:13.555890 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.555413 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 20:47:13.555988 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.555916 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 20:47:13.556332 ip-10-0-129-192 systemd[1]: Started Kubernetes Kubelet. Nov 03 20:47:13.557564 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.557545 2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 20:47:13.558031 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:13.558009 2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 20:47:13.560984 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.560969 2575 server.go:317] "Adding debug handlers to kubelet server" Nov 03 20:47:13.564982 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.564955 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 20:47:13.565591 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.565545 2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 20:47:13.566259 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:13.566051 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-192.us-west-2.compute.internal\" not found" Nov 03 20:47:13.566479 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.566451 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 20:47:13.566572 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.566514 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 20:47:13.566572 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.566532 2575 factory.go:55] Registering systemd factory Nov 03 20:47:13.566572 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.566540 2575 factory.go:223] Registration of the systemd container factory successfully Nov 03 20:47:13.566572 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.566571 2575 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 20:47:13.566728 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.566583 2575 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 20:47:13.566728 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.566682 2575 reconstruct.go:97] "Volume reconstruction finished" Nov 03 20:47:13.566728 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.566693 2575 reconciler.go:29] "Reconciler: start to sync state" Nov 03 20:47:13.567921 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.567903 2575 factory.go:153] Registering CRI-O factory Nov 03 20:47:13.568015 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.567925 2575 factory.go:223] Registration of the crio container factory successfully Nov 03 20:47:13.568015 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.567957 2575 factory.go:103] Registering Raw factory Nov 03 20:47:13.568015 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.567973 2575 manager.go:1199] Started watching for new ooms in manager Nov 03 20:47:13.569002 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.568986 2575 manager.go:319] Starting recovery of all containers Nov 03 20:47:13.581378 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.581359 2575 manager.go:324] Recovery completed Nov 03 20:47:13.585801 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.585785 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:47:13.587895 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.587877 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:47:13.587988 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.587912 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:47:13.587988 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.587925 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:47:13.588601 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.588586 2575 cpu_manager.go:226] "Starting" policy="none" Nov 03 20:47:13.588601 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.588596 2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 20:47:13.588697 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.588604 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:47:13.588697 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.588613 2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 20:47:13.589596 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.589581 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:47:13.589655 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.589607 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:47:13.589655 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.589617 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:47:13.589901 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.589889 2575 policy_none.go:50] "Start" Nov 03 20:47:13.589944 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.589906 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 20:47:13.589944 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.589916 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 20:47:13.591204 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.591192 2575 policy_none.go:44] "Start" Nov 03 20:47:13.630868 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.630690 2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 20:47:13.634897 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.634884 2575 manager.go:341] "Starting Device Plugin manager" Nov 03 20:47:13.634945 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:13.634914 2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 20:47:13.634945 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.634923 2575 server.go:87] "Starting device plugin registration server" Nov 03 20:47:13.635243 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.635230 2575 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 20:47:13.635294 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.635244 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 20:47:13.635377 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.635346 2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 20:47:13.635735 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.635468 2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 20:47:13.635735 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.635481 2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 20:47:13.636046 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:13.636019 2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 20:47:13.636114 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:13.636096 2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-192.us-west-2.compute.internal\" not found" Nov 03 20:47:13.687186 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:13.687144 2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-192.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 20:47:13.687186 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.687163 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-192.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 20:47:13.688569 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:13.687179 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-192.us-west-2.compute.internal.18e4a3f7299dfc7d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-192.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-192.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 20:47:13.550769277 +0000 UTC m=+0.412644696,LastTimestamp:2026-11-03 20:47:13.550769277 +0000 UTC m=+0.412644696,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-192.us-west-2.compute.internal,}" Nov 03 20:47:13.691173 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.691146 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 20:47:13.692451 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.692432 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 20:47:13.692451 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.692454 2575 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 20:47:13.692579 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.692480 2575 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 20:47:13.692579 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:13.692529 2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 20:47:13.735629 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.735547 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:47:13.736858 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.736838 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:47:13.736970 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.736869 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:47:13.736970 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.736881 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:47:13.736970 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.736905 2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:47:13.743172 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:13.743136 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 20:47:13.743308 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:13.743282 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 20:47:13.743378 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:13.743237 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-192.us-west-2.compute.internal.18e4a3f72bd48127 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-192.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-192.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-192.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 20:47:13.587896615 +0000 UTC m=+0.449772043,LastTimestamp:2026-11-03 20:47:13.587896615 +0000 UTC m=+0.449772043,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-192.us-west-2.compute.internal,}" Nov 03 20:47:13.754103 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.754066 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 20:47:13.754214 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.754158 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jjcgk" Nov 03 20:47:13.789022 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:13.788988 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-192.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 20:47:13.792881 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.792860 2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal"] Nov 03 20:47:13.792971 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.792953 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:47:13.793965 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.793943 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:47:13.794099 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.793977 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:47:13.794099 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.793991 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:47:13.795643 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.795620 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:47:13.795775 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.795758 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:47:13.795826 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.795791 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:47:13.796452 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.796434 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:47:13.796548 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.796466 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:47:13.796548 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.796477 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:47:13.796548 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.796440 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:47:13.796548 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.796542 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:47:13.796766 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.796555 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:47:13.798435 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.798420 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:47:13.798500 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.798446 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:47:13.799291 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.799273 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:47:13.799362 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.799302 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:47:13.799362 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.799318 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:47:13.824007 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:13.823978 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-192.us-west-2.compute.internal\" not found" node="ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:47:13.828225 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:13.828207 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-192.us-west-2.compute.internal\" not found" node="ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:47:13.838395 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.838375 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:47:13.846022 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.846004 2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:47:13.846129 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:13.846028 2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-192.us-west-2.compute.internal\": node \"ip-10-0-129-192.us-west-2.compute.internal\" not found" Nov 03 20:47:13.962381 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:13.962344 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-192.us-west-2.compute.internal\" not found" Nov 03 20:47:13.967548 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.967531 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/55575692b9132c5959a94942a022491a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal\" (UID: \"55575692b9132c5959a94942a022491a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:47:13.967629 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.967556 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55575692b9132c5959a94942a022491a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal\" (UID: \"55575692b9132c5959a94942a022491a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:47:13.967629 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:13.967581 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/448c85734c0da9850a3328446f82dce9-config\") pod \"kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal\" (UID: \"448c85734c0da9850a3328446f82dce9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:47:14.063855 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:14.063820 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-192.us-west-2.compute.internal\" not found" Nov 03 20:47:14.068626 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:14.068603 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55575692b9132c5959a94942a022491a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal\" (UID: \"55575692b9132c5959a94942a022491a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:47:14.068696 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:14.068638 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/448c85734c0da9850a3328446f82dce9-config\") pod \"kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal\" (UID: \"448c85734c0da9850a3328446f82dce9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:47:14.068696 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:14.068654 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/55575692b9132c5959a94942a022491a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal\" (UID: \"55575692b9132c5959a94942a022491a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:47:14.068807 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:14.068704 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/55575692b9132c5959a94942a022491a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal\" (UID: \"55575692b9132c5959a94942a022491a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:47:14.068807 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:14.068724 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55575692b9132c5959a94942a022491a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal\" (UID: \"55575692b9132c5959a94942a022491a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:47:14.068807 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:14.068746 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/448c85734c0da9850a3328446f82dce9-config\") pod \"kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal\" (UID: \"448c85734c0da9850a3328446f82dce9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:47:14.128585 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:14.128553 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:47:14.133693 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:14.132954 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:47:14.165626 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:14.165576 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-192.us-west-2.compute.internal\" not found" Nov 03 20:47:14.268017 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:14.267914 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-192.us-west-2.compute.internal\" not found" Nov 03 20:47:14.369191 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:14.369110 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-192.us-west-2.compute.internal\" not found" Nov 03 20:47:14.469735 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:14.469697 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-192.us-west-2.compute.internal\" not found" Nov 03 20:47:14.489350 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:14.489313 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 20:47:14.489537 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:14.489509 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 20:47:14.489608 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:14.489529 2575 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a7b0dd8fedf904086887a3a80115b944-867db6f534d550a3.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-129-192.us-west-2.compute.internal.18e4a3f72bd4d732\": read tcp 10.0.129.192:39410->16.58.146.143:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-129-192.us-west-2.compute.internal.18e4a3f72bd4d732 default 5441 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-192.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-192.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-192.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 20:47:13 +0000 UTC,LastTimestamp:2026-11-03 20:47:13.79398365 +0000 UTC m=+0.655859074,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-192.us-west-2.compute.internal,}" Nov 03 20:47:14.570014 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:14.569919 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-192.us-west-2.compute.internal\" not found" Nov 03 20:47:14.670385 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:14.670351 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-192.us-west-2.compute.internal\" not found" Nov 03 20:47:14.759898 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:14.759846 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 20:42:14 +0000 UTC" deadline="2028-07-15 02:31:05.748155724 +0000 UTC" Nov 03 20:47:14.759898 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:14.759885 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14861h43m50.988274177s" Nov 03 20:47:14.770958 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:14.770933 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-192.us-west-2.compute.internal\" not found" Nov 03 20:47:14.832569 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:14.832508 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:47:14.833570 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:14.833550 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:47:14.833647 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:14.833582 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:47:14.833647 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:14.833593 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:47:14.871705 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:14.871668 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-192.us-west-2.compute.internal\" not found" Nov 03 20:47:14.972361 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:14.972324 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-192.us-west-2.compute.internal\" not found" Nov 03 20:47:14.980499 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:14.980470 2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:47:14.980698 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:14.980680 2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:47:15.066883 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.066850 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:47:15.118891 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.118806 2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:47:15.133025 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.132998 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:47:15.133970 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.133950 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:47:15.200307 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.200275 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:47:15.451007 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:15.450965 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod448c85734c0da9850a3328446f82dce9.slice/crio-740a6d0a1a116d13d11958b6f022a8a647d699b659a1b9b83ee9a8acae3b8e22 WatchSource:0}: Error finding container 740a6d0a1a116d13d11958b6f022a8a647d699b659a1b9b83ee9a8acae3b8e22: Status 404 returned error can't find the container with id 740a6d0a1a116d13d11958b6f022a8a647d699b659a1b9b83ee9a8acae3b8e22 Nov 03 20:47:15.451331 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:15.451310 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55575692b9132c5959a94942a022491a.slice/crio-f60e0f8c9bd1538eeca368800ccd9da47cda90699ebc3e92d80a6a7b3ec38513 WatchSource:0}: Error finding container f60e0f8c9bd1538eeca368800ccd9da47cda90699ebc3e92d80a6a7b3ec38513: Status 404 returned error can't find the container with id f60e0f8c9bd1538eeca368800ccd9da47cda90699ebc3e92d80a6a7b3ec38513 Nov 03 20:47:15.457208 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.457187 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:47:15.538360 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.538327 2575 apiserver.go:52] "Watching apiserver" Nov 03 20:47:15.568811 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.568779 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 20:47:15.654565 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.654528 2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 20:47:15.654907 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.654883 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-qwqjp","openshift-dns/node-resolver-vq2rj","openshift-multus/multus-additional-cni-plugins-5p62l","openshift-multus/network-metrics-daemon-tqhnr","openshift-network-diagnostics/network-check-target-lm85p","openshift-network-operator/iptables-alerter-9bt29","openshift-ovn-kubernetes/ovnkube-node-mfbd2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc","openshift-image-registry/node-ca-lmq6t","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal","openshift-multus/multus-thxwg","kube-system/global-pull-secret-syncer-mqmds","kube-system/kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal","kube-system/konnectivity-agent-ksg72"] Nov 03 20:47:15.658218 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.658194 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.659732 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.659709 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.661388 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.661365 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:47:15.662745 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.662723 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:15.662847 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:15.662794 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqhnr" podUID="c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c" Nov 03 20:47:15.664373 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.664354 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:15.664477 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:15.664401 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85p" podUID="955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81" Nov 03 20:47:15.665960 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.665940 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9bt29" Nov 03 20:47:15.667563 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.667540 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lmq6t" Nov 03 20:47:15.669741 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.669723 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ksg72" Nov 03 20:47:15.673230 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.673214 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vq2rj" Nov 03 20:47:15.674810 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.674795 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:47:15.676477 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.676460 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.677783 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.677658 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-run-ovn\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.677783 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.677684 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-run-ovn-kubernetes\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.677783 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.677715 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-etc-kubernetes\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.677783 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.677744 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh5qw\" (UniqueName: \"kubernetes.io/projected/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-kube-api-access-dh5qw\") pod \"network-metrics-daemon-tqhnr\" (UID: \"c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c\") " pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:15.678028 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.677810 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5mgg\" (UniqueName: \"kubernetes.io/projected/955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81-kube-api-access-j5mgg\") pod \"network-check-target-lm85p\" (UID: \"955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81\") " pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:15.678028 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.677878 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-slash\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.678028 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.677914 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a2dc043c-52c0-4da0-8370-ed975ebb987f-ovnkube-config\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.678028 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.677941 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-run-netns\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.678028 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.677967 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-hostroot\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.678028 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.677993 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkxvg\" (UniqueName: \"kubernetes.io/projected/eaf3e4d3-7019-42e5-9e1e-2927b7f22304-kube-api-access-nkxvg\") pod \"iptables-alerter-9bt29\" (UID: \"eaf3e4d3-7019-42e5-9e1e-2927b7f22304\") " pod="openshift-network-operator/iptables-alerter-9bt29" Nov 03 20:47:15.678463 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.678042 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bfca9b06-e0d2-4d0f-a75c-7827485cb42d-host\") pod \"node-ca-lmq6t\" (UID: \"bfca9b06-e0d2-4d0f-a75c-7827485cb42d\") " pod="openshift-image-registry/node-ca-lmq6t" Nov 03 20:47:15.678463 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.678135 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-etc-openvswitch\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.678463 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.678167 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-cni-netd\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.678463 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.678227 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.678463 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.678273 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:15.678463 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.678275 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-cnibin\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.678463 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.678304 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-var-lib-kubelet\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.678463 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:15.678320 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqmds" podUID="712fcecb-4757-4dd0-8f96-084b1e6ee903" Nov 03 20:47:15.678463 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.678354 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-metrics-certs\") pod \"network-metrics-daemon-tqhnr\" (UID: \"c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c\") " pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:15.678463 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.678383 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a882e61f-2b83-4ff4-8370-cba401a6d9c8-cni-binary-copy\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:47:15.678463 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.678424 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-systemd-units\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.678463 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.678440 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-run-systemd\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.678463 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.678455 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2dc043c-52c0-4da0-8370-ed975ebb987f-ovn-node-metrics-cert\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.679117 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.678507 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-system-cni-dir\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.679117 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.678548 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eaf3e4d3-7019-42e5-9e1e-2927b7f22304-iptables-alerter-script\") pod \"iptables-alerter-9bt29\" (UID: \"eaf3e4d3-7019-42e5-9e1e-2927b7f22304\") " pod="openshift-network-operator/iptables-alerter-9bt29" Nov 03 20:47:15.679117 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.678586 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eaf3e4d3-7019-42e5-9e1e-2927b7f22304-host-slash\") pod \"iptables-alerter-9bt29\" (UID: \"eaf3e4d3-7019-42e5-9e1e-2927b7f22304\") " pod="openshift-network-operator/iptables-alerter-9bt29" Nov 03 20:47:15.679117 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.678627 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ttzr\" (UniqueName: \"kubernetes.io/projected/a882e61f-2b83-4ff4-8370-cba401a6d9c8-kube-api-access-6ttzr\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:47:15.679117 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.678658 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bfca9b06-e0d2-4d0f-a75c-7827485cb42d-serviceca\") pod \"node-ca-lmq6t\" (UID: \"bfca9b06-e0d2-4d0f-a75c-7827485cb42d\") " pod="openshift-image-registry/node-ca-lmq6t" Nov 03 20:47:15.679117 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.678686 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-kubelet\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.679117 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.678724 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-cni-bin\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.679117 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.678763 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-multus-conf-dir\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.679117 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.678787 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-run-multus-certs\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.679117 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.678815 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbctt\" (UniqueName: \"kubernetes.io/projected/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-kube-api-access-cbctt\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.679117 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.678842 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a882e61f-2b83-4ff4-8370-cba401a6d9c8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:47:15.679117 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.678894 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-log-socket\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.679117 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.678922 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42v78\" (UniqueName: \"kubernetes.io/projected/a2dc043c-52c0-4da0-8370-ed975ebb987f-kube-api-access-42v78\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.679117 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.678957 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-multus-cni-dir\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.679117 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.678983 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-var-lib-cni-bin\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.679117 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.679018 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a882e61f-2b83-4ff4-8370-cba401a6d9c8-system-cni-dir\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:47:15.679697 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.679063 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a882e61f-2b83-4ff4-8370-cba401a6d9c8-cnibin\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:47:15.679697 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.679101 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a882e61f-2b83-4ff4-8370-cba401a6d9c8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:47:15.679697 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.679120 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf5tz\" (UniqueName: \"kubernetes.io/projected/bfca9b06-e0d2-4d0f-a75c-7827485cb42d-kube-api-access-tf5tz\") pod \"node-ca-lmq6t\" (UID: \"bfca9b06-e0d2-4d0f-a75c-7827485cb42d\") " pod="openshift-image-registry/node-ca-lmq6t" Nov 03 20:47:15.679697 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.679135 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-run-netns\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.679697 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.679155 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-node-log\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.679697 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.679186 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a2dc043c-52c0-4da0-8370-ed975ebb987f-env-overrides\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.679697 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.679235 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a2dc043c-52c0-4da0-8370-ed975ebb987f-ovnkube-script-lib\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.679697 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.679292 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-cni-binary-copy\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.679697 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.679323 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-multus-socket-dir-parent\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.679697 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.679356 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-run-k8s-cni-cncf-io\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.679697 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.679377 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-var-lib-cni-multus\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.679697 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.679410 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-multus-daemon-config\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.679697 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.679466 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a882e61f-2b83-4ff4-8370-cba401a6d9c8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:47:15.679697 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.679504 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-var-lib-openvswitch\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.679697 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.679530 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-run-openvswitch\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.697484 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.697427 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal" event={"ID":"55575692b9132c5959a94942a022491a","Type":"ContainerStarted","Data":"f60e0f8c9bd1538eeca368800ccd9da47cda90699ebc3e92d80a6a7b3ec38513"} Nov 03 20:47:15.698858 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.698832 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal" event={"ID":"448c85734c0da9850a3328446f82dce9","Type":"ContainerStarted","Data":"740a6d0a1a116d13d11958b6f022a8a647d699b659a1b9b83ee9a8acae3b8e22"} Nov 03 20:47:15.718454 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.718432 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 20:47:15.718584 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.718535 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 20:47:15.718659 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.718607 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 20:47:15.718659 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.718637 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 20:47:15.718769 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.718666 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 20:47:15.718769 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.718642 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xldt8\"" Nov 03 20:47:15.718769 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.718698 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 20:47:15.718769 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.718708 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 20:47:15.718769 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.718709 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 20:47:15.718769 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.718756 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 20:47:15.718769 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.718766 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ldspv\"" Nov 03 20:47:15.719046 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.719030 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 20:47:15.719807 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.719788 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-r6str\"" Nov 03 20:47:15.769212 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.769180 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 20:47:15.771314 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.771293 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 20:47:15.771409 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.771339 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tp4fx" Nov 03 20:47:15.780785 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.780754 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-cnibin\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.780888 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.780793 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-var-lib-kubelet\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.780888 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.780819 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-metrics-certs\") pod \"network-metrics-daemon-tqhnr\" (UID: \"c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c\") " pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:15.780888 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.780850 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prrnl\" (UniqueName: \"kubernetes.io/projected/a3e654a3-7d76-4793-9627-6334ba6db6d9-kube-api-access-prrnl\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:47:15.780888 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.780870 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-var-lib-kubelet\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.781071 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.780931 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-systemd-units\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.781071 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.780874 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-cnibin\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.781071 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.780879 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-systemd-units\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.781071 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:15.780947 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:47:15.781071 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.780976 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-system-cni-dir\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.781071 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.780999 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ttzr\" (UniqueName: \"kubernetes.io/projected/a882e61f-2b83-4ff4-8370-cba401a6d9c8-kube-api-access-6ttzr\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:47:15.781071 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781015 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-cni-bin\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.781071 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781030 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-cni-bin\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.781071 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:15.781043 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-metrics-certs podName:c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c nodeName:}" failed. No retries permitted until 2026-11-03 20:47:16.281019631 +0000 UTC m=+3.142895058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-metrics-certs") pod "network-metrics-daemon-tqhnr" (UID: "c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:47:15.781071 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781061 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-system-cni-dir\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.781509 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781100 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-multus-conf-dir\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.781509 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781123 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-multus-conf-dir\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.781509 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781133 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-run-multus-certs\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.781509 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781161 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a882e61f-2b83-4ff4-8370-cba401a6d9c8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:47:15.781509 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781185 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-run-multus-certs\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.781509 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781191 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-sysctl-d\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.781509 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781281 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/712fcecb-4757-4dd0-8f96-084b1e6ee903-original-pull-secret\") pod \"global-pull-secret-syncer-mqmds\" (UID: \"712fcecb-4757-4dd0-8f96-084b1e6ee903\") " pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:15.781509 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781318 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-log-socket\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.781509 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781338 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42v78\" (UniqueName: \"kubernetes.io/projected/a2dc043c-52c0-4da0-8370-ed975ebb987f-kube-api-access-42v78\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.781509 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781362 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a882e61f-2b83-4ff4-8370-cba401a6d9c8-system-cni-dir\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:47:15.781509 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781379 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a882e61f-2b83-4ff4-8370-cba401a6d9c8-cnibin\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:47:15.781509 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781397 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a882e61f-2b83-4ff4-8370-cba401a6d9c8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:47:15.781509 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781362 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-log-socket\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.781509 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781422 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:47:15.781509 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781447 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/026b5dc8-bcff-4afc-a784-baf95c1cec27-tmp\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.781509 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781471 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-node-log\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.781509 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781476 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a882e61f-2b83-4ff4-8370-cba401a6d9c8-cnibin\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:47:15.782359 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781481 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a882e61f-2b83-4ff4-8370-cba401a6d9c8-system-cni-dir\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:47:15.782359 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781513 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-node-log\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.782359 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781540 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a2dc043c-52c0-4da0-8370-ed975ebb987f-env-overrides\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.782359 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781556 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-run-k8s-cni-cncf-io\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.782359 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781612 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a882e61f-2b83-4ff4-8370-cba401a6d9c8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:47:15.782359 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781614 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-run-k8s-cni-cncf-io\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.782359 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781660 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-multus-daemon-config\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.782359 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781705 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cba96b6a-ccc8-4c1c-b371-1bb5ece29afc-konnectivity-ca\") pod \"konnectivity-agent-ksg72\" (UID: \"cba96b6a-ccc8-4c1c-b371-1bb5ece29afc\") " pod="kube-system/konnectivity-agent-ksg72" Nov 03 20:47:15.782359 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781752 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:47:15.782359 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781784 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-lib-modules\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.782359 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781814 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-var-lib-openvswitch\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.782359 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781840 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-run-ovn-kubernetes\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.782359 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781849 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a882e61f-2b83-4ff4-8370-cba401a6d9c8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:47:15.782359 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781867 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-etc-kubernetes\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.782359 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781900 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-var-lib-openvswitch\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.782359 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781909 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-run-ovn-kubernetes\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.782359 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781919 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-etc-kubernetes\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.783114 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781929 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh5qw\" (UniqueName: \"kubernetes.io/projected/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-kube-api-access-dh5qw\") pod \"network-metrics-daemon-tqhnr\" (UID: \"c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c\") " pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:15.783114 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781956 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-socket-dir\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:47:15.783114 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781974 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/712fcecb-4757-4dd0-8f96-084b1e6ee903-kubelet-config\") pod \"global-pull-secret-syncer-mqmds\" (UID: \"712fcecb-4757-4dd0-8f96-084b1e6ee903\") " pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:15.783114 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.781992 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-slash\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.783114 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782028 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-slash\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.783114 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782044 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a2dc043c-52c0-4da0-8370-ed975ebb987f-env-overrides\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.783114 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782044 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a2dc043c-52c0-4da0-8370-ed975ebb987f-ovnkube-config\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.783114 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782081 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-run-netns\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.783114 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782108 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-hostroot\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.783114 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782132 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-run-netns\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.783114 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782146 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-registration-dir\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:47:15.783114 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782165 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv6zc\" (UniqueName: \"kubernetes.io/projected/9050192a-0ec2-45b9-8b2c-2beb0501b09b-kube-api-access-tv6zc\") pod \"node-resolver-vq2rj\" (UID: \"9050192a-0ec2-45b9-8b2c-2beb0501b09b\") " pod="openshift-dns/node-resolver-vq2rj" Nov 03 20:47:15.783114 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782186 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-tuned\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.783114 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782194 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-hostroot\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.783114 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782231 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-etc-openvswitch\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.783114 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782251 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-cni-netd\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.783114 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782267 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-etc-openvswitch\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.783766 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782283 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-multus-daemon-config\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.783766 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782298 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-cni-netd\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.783766 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782309 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.783766 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782341 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a882e61f-2b83-4ff4-8370-cba401a6d9c8-cni-binary-copy\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:47:15.783766 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782356 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.783766 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782367 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bfca9b06-e0d2-4d0f-a75c-7827485cb42d-serviceca\") pod \"node-ca-lmq6t\" (UID: \"bfca9b06-e0d2-4d0f-a75c-7827485cb42d\") " pod="openshift-image-registry/node-ca-lmq6t" Nov 03 20:47:15.783766 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782394 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-run-systemd\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.783766 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782420 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2dc043c-52c0-4da0-8370-ed975ebb987f-ovn-node-metrics-cert\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.783766 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782446 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eaf3e4d3-7019-42e5-9e1e-2927b7f22304-iptables-alerter-script\") pod \"iptables-alerter-9bt29\" (UID: \"eaf3e4d3-7019-42e5-9e1e-2927b7f22304\") " pod="openshift-network-operator/iptables-alerter-9bt29" Nov 03 20:47:15.783766 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782461 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-run-systemd\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.783766 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782473 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eaf3e4d3-7019-42e5-9e1e-2927b7f22304-host-slash\") pod \"iptables-alerter-9bt29\" (UID: \"eaf3e4d3-7019-42e5-9e1e-2927b7f22304\") " pod="openshift-network-operator/iptables-alerter-9bt29" Nov 03 20:47:15.783766 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782511 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-kubelet\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.783766 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782525 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eaf3e4d3-7019-42e5-9e1e-2927b7f22304-host-slash\") pod \"iptables-alerter-9bt29\" (UID: \"eaf3e4d3-7019-42e5-9e1e-2927b7f22304\") " pod="openshift-network-operator/iptables-alerter-9bt29" Nov 03 20:47:15.783766 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782537 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbctt\" (UniqueName: \"kubernetes.io/projected/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-kube-api-access-cbctt\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.783766 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782547 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a2dc043c-52c0-4da0-8370-ed975ebb987f-ovnkube-config\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.783766 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782573 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-kubelet\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.783766 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782564 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-device-dir\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:47:15.784322 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782612 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9050192a-0ec2-45b9-8b2c-2beb0501b09b-hosts-file\") pod \"node-resolver-vq2rj\" (UID: \"9050192a-0ec2-45b9-8b2c-2beb0501b09b\") " pod="openshift-dns/node-resolver-vq2rj" Nov 03 20:47:15.784322 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782721 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9050192a-0ec2-45b9-8b2c-2beb0501b09b-tmp-dir\") pod \"node-resolver-vq2rj\" (UID: \"9050192a-0ec2-45b9-8b2c-2beb0501b09b\") " pod="openshift-dns/node-resolver-vq2rj" Nov 03 20:47:15.784322 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782748 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-sysctl-conf\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.784322 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782784 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-systemd\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.784322 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782782 2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 20:47:15.784322 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782833 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-run\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.784322 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782849 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a882e61f-2b83-4ff4-8370-cba401a6d9c8-cni-binary-copy\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:47:15.784322 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782879 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-multus-cni-dir\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.784322 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782915 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-var-lib-cni-bin\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.784322 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782942 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf5tz\" (UniqueName: \"kubernetes.io/projected/bfca9b06-e0d2-4d0f-a75c-7827485cb42d-kube-api-access-tf5tz\") pod \"node-ca-lmq6t\" (UID: \"bfca9b06-e0d2-4d0f-a75c-7827485cb42d\") " pod="openshift-image-registry/node-ca-lmq6t" Nov 03 20:47:15.784322 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.782981 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-var-lib-cni-bin\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.784322 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.783014 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-run-netns\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.784322 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.783046 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a2dc043c-52c0-4da0-8370-ed975ebb987f-ovnkube-script-lib\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.784322 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.783067 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-multus-cni-dir\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.784322 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.783066 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-run-netns\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.784322 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.783071 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-cni-binary-copy\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.784322 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.783141 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-multus-socket-dir-parent\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.784322 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.783167 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-var-lib-cni-multus\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.784848 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.783217 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-sys-fs\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:47:15.784848 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.783223 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-multus-socket-dir-parent\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.784848 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.783244 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-var-lib-cni-multus\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.784848 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.783247 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cba96b6a-ccc8-4c1c-b371-1bb5ece29afc-agent-certs\") pod \"konnectivity-agent-ksg72\" (UID: \"cba96b6a-ccc8-4c1c-b371-1bb5ece29afc\") " pod="kube-system/konnectivity-agent-ksg72" Nov 03 20:47:15.784848 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.783291 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-sysconfig\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.784848 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.783325 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-sys\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.784848 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.783363 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-host\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.784848 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.783391 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-run-openvswitch\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.784848 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.783417 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-run-ovn\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.784848 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.783445 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5mgg\" (UniqueName: \"kubernetes.io/projected/955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81-kube-api-access-j5mgg\") pod \"network-check-target-lm85p\" (UID: \"955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81\") " pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:15.784848 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.783494 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-run-ovn\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.784848 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.783524 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkxvg\" (UniqueName: \"kubernetes.io/projected/eaf3e4d3-7019-42e5-9e1e-2927b7f22304-kube-api-access-nkxvg\") pod \"iptables-alerter-9bt29\" (UID: \"eaf3e4d3-7019-42e5-9e1e-2927b7f22304\") " pod="openshift-network-operator/iptables-alerter-9bt29" Nov 03 20:47:15.784848 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.783551 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bfca9b06-e0d2-4d0f-a75c-7827485cb42d-host\") pod \"node-ca-lmq6t\" (UID: \"bfca9b06-e0d2-4d0f-a75c-7827485cb42d\") " pod="openshift-image-registry/node-ca-lmq6t" Nov 03 20:47:15.784848 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.783556 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-cni-binary-copy\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.784848 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.783579 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-modprobe-d\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.784848 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.783607 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-kubernetes\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.784848 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.783617 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a2dc043c-52c0-4da0-8370-ed975ebb987f-ovnkube-script-lib\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.785364 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.783634 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-var-lib-kubelet\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.785364 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.783667 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-run-openvswitch\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.785364 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.783692 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bfca9b06-e0d2-4d0f-a75c-7827485cb42d-host\") pod \"node-ca-lmq6t\" (UID: \"bfca9b06-e0d2-4d0f-a75c-7827485cb42d\") " pod="openshift-image-registry/node-ca-lmq6t" Nov 03 20:47:15.785364 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.783724 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6lpt\" (UniqueName: \"kubernetes.io/projected/026b5dc8-bcff-4afc-a784-baf95c1cec27-kube-api-access-p6lpt\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.785364 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.783772 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/712fcecb-4757-4dd0-8f96-084b1e6ee903-dbus\") pod \"global-pull-secret-syncer-mqmds\" (UID: \"712fcecb-4757-4dd0-8f96-084b1e6ee903\") " pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:15.786477 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.786456 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2dc043c-52c0-4da0-8370-ed975ebb987f-ovn-node-metrics-cert\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.790768 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.790751 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:47:15.826957 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.826923 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 20:47:15.827120 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.827035 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 20:47:15.827191 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.827115 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 20:47:15.827267 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.827203 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 20:47:15.827267 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.827228 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 20:47:15.827351 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.827268 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 20:47:15.827351 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.827339 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 20:47:15.827479 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.827461 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-868xx\"" Nov 03 20:47:15.827525 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.827513 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 20:47:15.827582 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.827565 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 20:47:15.827620 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.827562 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 20:47:15.827720 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.827707 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:47:15.827768 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.827726 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 20:47:15.827768 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.827732 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 20:47:15.827768 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.827765 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:47:15.827915 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.827784 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 20:47:15.827915 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.827866 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 20:47:15.827915 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.827876 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2g8j2\"" Nov 03 20:47:15.827915 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.827879 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 20:47:15.827915 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.827905 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gf6gt\"" Nov 03 20:47:15.828074 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.827915 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vwxfp\"" Nov 03 20:47:15.828074 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.827923 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7vv6z\"" Nov 03 20:47:15.828074 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.827928 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tlf9q\"" Nov 03 20:47:15.832860 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.832842 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a882e61f-2b83-4ff4-8370-cba401a6d9c8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:47:15.832925 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.832905 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bfca9b06-e0d2-4d0f-a75c-7827485cb42d-serviceca\") pod \"node-ca-lmq6t\" (UID: \"bfca9b06-e0d2-4d0f-a75c-7827485cb42d\") " pod="openshift-image-registry/node-ca-lmq6t" Nov 03 20:47:15.832963 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.832930 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a882e61f-2b83-4ff4-8370-cba401a6d9c8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:47:15.833803 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.833787 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eaf3e4d3-7019-42e5-9e1e-2927b7f22304-iptables-alerter-script\") pod \"iptables-alerter-9bt29\" (UID: \"eaf3e4d3-7019-42e5-9e1e-2927b7f22304\") " pod="openshift-network-operator/iptables-alerter-9bt29" Nov 03 20:47:15.848424 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:15.848398 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:47:15.848424 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:15.848426 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:47:15.848648 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:15.848441 2575 projected.go:196] Error preparing data for projected volume kube-api-access-j5mgg for pod openshift-network-diagnostics/network-check-target-lm85p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:47:15.848648 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:15.848526 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81-kube-api-access-j5mgg podName:955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:16.348503989 +0000 UTC m=+3.210379416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-j5mgg" (UniqueName: "kubernetes.io/projected/955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81-kube-api-access-j5mgg") pod "network-check-target-lm85p" (UID: "955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:47:15.849707 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.849677 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ttzr\" (UniqueName: \"kubernetes.io/projected/a882e61f-2b83-4ff4-8370-cba401a6d9c8-kube-api-access-6ttzr\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:47:15.849874 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.849847 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42v78\" (UniqueName: \"kubernetes.io/projected/a2dc043c-52c0-4da0-8370-ed975ebb987f-kube-api-access-42v78\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.849963 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.849879 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbctt\" (UniqueName: \"kubernetes.io/projected/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-kube-api-access-cbctt\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.849963 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.849878 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh5qw\" (UniqueName: \"kubernetes.io/projected/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-kube-api-access-dh5qw\") pod \"network-metrics-daemon-tqhnr\" (UID: \"c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c\") " pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:15.850687 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.850669 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf5tz\" (UniqueName: \"kubernetes.io/projected/bfca9b06-e0d2-4d0f-a75c-7827485cb42d-kube-api-access-tf5tz\") pod \"node-ca-lmq6t\" (UID: \"bfca9b06-e0d2-4d0f-a75c-7827485cb42d\") " pod="openshift-image-registry/node-ca-lmq6t" Nov 03 20:47:15.851090 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.851073 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkxvg\" (UniqueName: \"kubernetes.io/projected/eaf3e4d3-7019-42e5-9e1e-2927b7f22304-kube-api-access-nkxvg\") pod \"iptables-alerter-9bt29\" (UID: \"eaf3e4d3-7019-42e5-9e1e-2927b7f22304\") " pod="openshift-network-operator/iptables-alerter-9bt29" Nov 03 20:47:15.884597 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.884568 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-tuned\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.884702 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.884605 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-device-dir\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:47:15.884702 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.884622 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9050192a-0ec2-45b9-8b2c-2beb0501b09b-hosts-file\") pod \"node-resolver-vq2rj\" (UID: \"9050192a-0ec2-45b9-8b2c-2beb0501b09b\") " pod="openshift-dns/node-resolver-vq2rj" Nov 03 20:47:15.884702 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.884682 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9050192a-0ec2-45b9-8b2c-2beb0501b09b-tmp-dir\") pod \"node-resolver-vq2rj\" (UID: \"9050192a-0ec2-45b9-8b2c-2beb0501b09b\") " pod="openshift-dns/node-resolver-vq2rj" Nov 03 20:47:15.884833 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.884703 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9050192a-0ec2-45b9-8b2c-2beb0501b09b-hosts-file\") pod \"node-resolver-vq2rj\" (UID: \"9050192a-0ec2-45b9-8b2c-2beb0501b09b\") " pod="openshift-dns/node-resolver-vq2rj" Nov 03 20:47:15.884833 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.884711 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-sysctl-conf\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.884833 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.884736 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-systemd\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.884833 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.884772 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-run\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.884833 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.884785 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-device-dir\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:47:15.884833 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.884799 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-sys-fs\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:47:15.884833 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.884820 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cba96b6a-ccc8-4c1c-b371-1bb5ece29afc-agent-certs\") pod \"konnectivity-agent-ksg72\" (UID: \"cba96b6a-ccc8-4c1c-b371-1bb5ece29afc\") " pod="kube-system/konnectivity-agent-ksg72" Nov 03 20:47:15.884833 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.884835 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-sysconfig\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.885221 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.884840 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-systemd\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.885221 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.884859 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-sys\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.885221 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.884867 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-sys-fs\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:47:15.885221 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.884849 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-sysctl-conf\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.885221 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.884899 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-run\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.885221 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.884892 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-host\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.885221 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.884913 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-sys\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.885221 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.884935 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-host\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.885221 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.884961 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-modprobe-d\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.885221 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.884986 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-kubernetes\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.885221 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.885006 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-var-lib-kubelet\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.885221 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.884962 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-sysconfig\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.885221 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.885058 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6lpt\" (UniqueName: \"kubernetes.io/projected/026b5dc8-bcff-4afc-a784-baf95c1cec27-kube-api-access-p6lpt\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.885221 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.885075 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/712fcecb-4757-4dd0-8f96-084b1e6ee903-dbus\") pod \"global-pull-secret-syncer-mqmds\" (UID: \"712fcecb-4757-4dd0-8f96-084b1e6ee903\") " pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:15.885221 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.885105 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-kubernetes\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.885221 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.885116 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-modprobe-d\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.885943 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.885504 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prrnl\" (UniqueName: \"kubernetes.io/projected/a3e654a3-7d76-4793-9627-6334ba6db6d9-kube-api-access-prrnl\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:47:15.885943 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.885581 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-sysctl-d\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.885943 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.885617 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/712fcecb-4757-4dd0-8f96-084b1e6ee903-original-pull-secret\") pod \"global-pull-secret-syncer-mqmds\" (UID: \"712fcecb-4757-4dd0-8f96-084b1e6ee903\") " pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:15.885943 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.885664 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:47:15.885943 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.885709 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/026b5dc8-bcff-4afc-a784-baf95c1cec27-tmp\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.885943 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.885758 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cba96b6a-ccc8-4c1c-b371-1bb5ece29afc-konnectivity-ca\") pod \"konnectivity-agent-ksg72\" (UID: \"cba96b6a-ccc8-4c1c-b371-1bb5ece29afc\") " pod="kube-system/konnectivity-agent-ksg72" Nov 03 20:47:15.885943 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.885788 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:47:15.885943 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.885823 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-lib-modules\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.885943 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.885860 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9050192a-0ec2-45b9-8b2c-2beb0501b09b-tmp-dir\") pod \"node-resolver-vq2rj\" (UID: \"9050192a-0ec2-45b9-8b2c-2beb0501b09b\") " pod="openshift-dns/node-resolver-vq2rj" Nov 03 20:47:15.885943 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.885505 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-var-lib-kubelet\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.886520 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.886024 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-sysctl-d\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.886520 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.886100 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-socket-dir\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:47:15.886520 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.885866 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-socket-dir\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:47:15.886520 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.886190 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:47:15.886520 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.886222 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/712fcecb-4757-4dd0-8f96-084b1e6ee903-kubelet-config\") pod \"global-pull-secret-syncer-mqmds\" (UID: \"712fcecb-4757-4dd0-8f96-084b1e6ee903\") " pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:15.886520 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.886296 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/712fcecb-4757-4dd0-8f96-084b1e6ee903-kubelet-config\") pod \"global-pull-secret-syncer-mqmds\" (UID: \"712fcecb-4757-4dd0-8f96-084b1e6ee903\") " pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:15.886520 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.886298 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-lib-modules\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.886520 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.886351 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:47:15.886520 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.886366 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/712fcecb-4757-4dd0-8f96-084b1e6ee903-dbus\") pod \"global-pull-secret-syncer-mqmds\" (UID: \"712fcecb-4757-4dd0-8f96-084b1e6ee903\") " pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:15.886520 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.886428 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-registration-dir\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:47:15.886520 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.886477 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-registration-dir\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:47:15.886520 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.886514 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv6zc\" (UniqueName: \"kubernetes.io/projected/9050192a-0ec2-45b9-8b2c-2beb0501b09b-kube-api-access-tv6zc\") pod \"node-resolver-vq2rj\" (UID: \"9050192a-0ec2-45b9-8b2c-2beb0501b09b\") " pod="openshift-dns/node-resolver-vq2rj" Nov 03 20:47:15.887175 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:15.886817 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:47:15.887175 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:15.886894 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/712fcecb-4757-4dd0-8f96-084b1e6ee903-original-pull-secret podName:712fcecb-4757-4dd0-8f96-084b1e6ee903 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:16.386874716 +0000 UTC m=+3.248750138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/712fcecb-4757-4dd0-8f96-084b1e6ee903-original-pull-secret") pod "global-pull-secret-syncer-mqmds" (UID: "712fcecb-4757-4dd0-8f96-084b1e6ee903") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:47:15.887594 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.887567 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cba96b6a-ccc8-4c1c-b371-1bb5ece29afc-konnectivity-ca\") pod \"konnectivity-agent-ksg72\" (UID: \"cba96b6a-ccc8-4c1c-b371-1bb5ece29afc\") " pod="kube-system/konnectivity-agent-ksg72" Nov 03 20:47:15.887765 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.887744 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-tuned\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.888002 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.887982 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cba96b6a-ccc8-4c1c-b371-1bb5ece29afc-agent-certs\") pod \"konnectivity-agent-ksg72\" (UID: \"cba96b6a-ccc8-4c1c-b371-1bb5ece29afc\") " pod="kube-system/konnectivity-agent-ksg72" Nov 03 20:47:15.889984 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.889958 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/026b5dc8-bcff-4afc-a784-baf95c1cec27-tmp\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.954012 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.953938 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv6zc\" (UniqueName: \"kubernetes.io/projected/9050192a-0ec2-45b9-8b2c-2beb0501b09b-kube-api-access-tv6zc\") pod \"node-resolver-vq2rj\" (UID: \"9050192a-0ec2-45b9-8b2c-2beb0501b09b\") " pod="openshift-dns/node-resolver-vq2rj" Nov 03 20:47:15.954012 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.953940 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6lpt\" (UniqueName: \"kubernetes.io/projected/026b5dc8-bcff-4afc-a784-baf95c1cec27-kube-api-access-p6lpt\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:15.954170 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.954015 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prrnl\" (UniqueName: \"kubernetes.io/projected/a3e654a3-7d76-4793-9627-6334ba6db6d9-kube-api-access-prrnl\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:47:15.980720 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.980693 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-thxwg" Nov 03 20:47:15.984012 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.983992 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:15.991667 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:15.991640 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd8ecac_e0b7_4453_a4b5_c17ba6504cbb.slice/crio-30c9d23cea06d2e2ee7e26b675a4eadcd7b354fc941609a1dcae96f319a95d40 WatchSource:0}: Error finding container 30c9d23cea06d2e2ee7e26b675a4eadcd7b354fc941609a1dcae96f319a95d40: Status 404 returned error can't find the container with id 30c9d23cea06d2e2ee7e26b675a4eadcd7b354fc941609a1dcae96f319a95d40 Nov 03 20:47:15.994638 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:15.994615 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2dc043c_52c0_4da0_8370_ed975ebb987f.slice/crio-546bdb5314f9cefee256c637fee36db01e128465d8deb23c89c9021544635501 WatchSource:0}: Error finding container 546bdb5314f9cefee256c637fee36db01e128465d8deb23c89c9021544635501: Status 404 returned error can't find the container with id 546bdb5314f9cefee256c637fee36db01e128465d8deb23c89c9021544635501 Nov 03 20:47:15.998239 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.998220 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9bt29" Nov 03 20:47:15.998239 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:15.998235 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lmq6t" Nov 03 20:47:16.001970 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:16.001951 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:47:16.003287 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:16.003262 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ksg72" Nov 03 20:47:16.008056 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:16.008031 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf3e4d3_7019_42e5_9e1e_2927b7f22304.slice/crio-0d1c0d124ccb983ea54849dfaf3549fc40b31296c38641a01591988123f40fc5 WatchSource:0}: Error finding container 0d1c0d124ccb983ea54849dfaf3549fc40b31296c38641a01591988123f40fc5: Status 404 returned error can't find the container with id 0d1c0d124ccb983ea54849dfaf3549fc40b31296c38641a01591988123f40fc5 Nov 03 20:47:16.009195 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:16.009173 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vq2rj" Nov 03 20:47:16.012633 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:16.012609 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfca9b06_e0d2_4d0f_a75c_7827485cb42d.slice/crio-14a9ff28a6078cfcb41c163e90a0d9ba2abb7f39cf9fe994bc2a0cbc5e5b6cb0 WatchSource:0}: Error finding container 14a9ff28a6078cfcb41c163e90a0d9ba2abb7f39cf9fe994bc2a0cbc5e5b6cb0: Status 404 returned error can't find the container with id 14a9ff28a6078cfcb41c163e90a0d9ba2abb7f39cf9fe994bc2a0cbc5e5b6cb0 Nov 03 20:47:16.017066 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:16.017041 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba96b6a_ccc8_4c1c_b371_1bb5ece29afc.slice/crio-95a33df2d7282b2989b0a95d55a9d31eb9be8940fab157789f5a8e3fe0414f91 WatchSource:0}: Error finding container 95a33df2d7282b2989b0a95d55a9d31eb9be8940fab157789f5a8e3fe0414f91: Status 404 returned error can't find the container with id 95a33df2d7282b2989b0a95d55a9d31eb9be8940fab157789f5a8e3fe0414f91 Nov 03 20:47:16.017819 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:16.017754 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda882e61f_2b83_4ff4_8370_cba401a6d9c8.slice/crio-a2629873f88b6fb29b09dd4a207c332ea166c13a739da8c750cf06f33bee101e WatchSource:0}: Error finding container a2629873f88b6fb29b09dd4a207c332ea166c13a739da8c750cf06f33bee101e: Status 404 returned error can't find the container with id a2629873f88b6fb29b09dd4a207c332ea166c13a739da8c750cf06f33bee101e Nov 03 20:47:16.018999 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:16.018946 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:47:16.021265 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:16.021247 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9050192a_0ec2_45b9_8b2c_2beb0501b09b.slice/crio-8511e8faa04dfe9c6a804c5e3b4990480722ceaed7d8012f4667c5410dd8332a WatchSource:0}: Error finding container 8511e8faa04dfe9c6a804c5e3b4990480722ceaed7d8012f4667c5410dd8332a: Status 404 returned error can't find the container with id 8511e8faa04dfe9c6a804c5e3b4990480722ceaed7d8012f4667c5410dd8332a Nov 03 20:47:16.021499 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:16.021478 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:47:16.029279 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:16.029255 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e654a3_7d76_4793_9627_6334ba6db6d9.slice/crio-a2a172ce3de787560a04bafaebe785f49aebf30c6a2179418bd1dcef7cea3328 WatchSource:0}: Error finding container a2a172ce3de787560a04bafaebe785f49aebf30c6a2179418bd1dcef7cea3328: Status 404 returned error can't find the container with id a2a172ce3de787560a04bafaebe785f49aebf30c6a2179418bd1dcef7cea3328 Nov 03 20:47:16.029891 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:16.029867 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod026b5dc8_bcff_4afc_a784_baf95c1cec27.slice/crio-1159c0e719ea9bc23cc3be891067c9d285f5e8054a74a5b78394680ab2814563 WatchSource:0}: Error finding container 1159c0e719ea9bc23cc3be891067c9d285f5e8054a74a5b78394680ab2814563: Status 404 returned error can't find the container with id 1159c0e719ea9bc23cc3be891067c9d285f5e8054a74a5b78394680ab2814563 Nov 03 20:47:16.290518 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:16.290430 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-metrics-certs\") pod \"network-metrics-daemon-tqhnr\" (UID: \"c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c\") " pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:16.290668 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:16.290561 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:47:16.290668 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:16.290616 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-metrics-certs podName:c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c nodeName:}" failed. No retries permitted until 2026-11-03 20:47:17.290602599 +0000 UTC m=+4.152478021 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-metrics-certs") pod "network-metrics-daemon-tqhnr" (UID: "c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:47:16.390849 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:16.390808 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5mgg\" (UniqueName: \"kubernetes.io/projected/955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81-kube-api-access-j5mgg\") pod \"network-check-target-lm85p\" (UID: \"955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81\") " pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:16.391000 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:16.390955 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/712fcecb-4757-4dd0-8f96-084b1e6ee903-original-pull-secret\") pod \"global-pull-secret-syncer-mqmds\" (UID: \"712fcecb-4757-4dd0-8f96-084b1e6ee903\") " pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:16.391000 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:16.390982 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:47:16.391000 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:16.391000 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:47:16.391143 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:16.391016 2575 projected.go:196] Error preparing data for projected volume kube-api-access-j5mgg for pod openshift-network-diagnostics/network-check-target-lm85p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:47:16.391143 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:16.391065 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:47:16.391143 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:16.391076 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81-kube-api-access-j5mgg podName:955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:17.39106162 +0000 UTC m=+4.252937029 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-j5mgg" (UniqueName: "kubernetes.io/projected/955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81-kube-api-access-j5mgg") pod "network-check-target-lm85p" (UID: "955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:47:16.391143 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:16.391122 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/712fcecb-4757-4dd0-8f96-084b1e6ee903-original-pull-secret podName:712fcecb-4757-4dd0-8f96-084b1e6ee903 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:17.391108514 +0000 UTC m=+4.252983935 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/712fcecb-4757-4dd0-8f96-084b1e6ee903-original-pull-secret") pod "global-pull-secret-syncer-mqmds" (UID: "712fcecb-4757-4dd0-8f96-084b1e6ee903") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:47:16.702430 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:16.702330 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" event={"ID":"026b5dc8-bcff-4afc-a784-baf95c1cec27","Type":"ContainerStarted","Data":"1159c0e719ea9bc23cc3be891067c9d285f5e8054a74a5b78394680ab2814563"} Nov 03 20:47:16.704660 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:16.704625 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" event={"ID":"a3e654a3-7d76-4793-9627-6334ba6db6d9","Type":"ContainerStarted","Data":"a2a172ce3de787560a04bafaebe785f49aebf30c6a2179418bd1dcef7cea3328"} Nov 03 20:47:16.706089 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:16.706058 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vq2rj" event={"ID":"9050192a-0ec2-45b9-8b2c-2beb0501b09b","Type":"ContainerStarted","Data":"8511e8faa04dfe9c6a804c5e3b4990480722ceaed7d8012f4667c5410dd8332a"} Nov 03 20:47:16.707396 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:16.707364 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ksg72" event={"ID":"cba96b6a-ccc8-4c1c-b371-1bb5ece29afc","Type":"ContainerStarted","Data":"95a33df2d7282b2989b0a95d55a9d31eb9be8940fab157789f5a8e3fe0414f91"} Nov 03 20:47:16.708751 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:16.708721 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5p62l" event={"ID":"a882e61f-2b83-4ff4-8370-cba401a6d9c8","Type":"ContainerStarted","Data":"a2629873f88b6fb29b09dd4a207c332ea166c13a739da8c750cf06f33bee101e"} Nov 03 20:47:16.710127 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:16.710100 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lmq6t" event={"ID":"bfca9b06-e0d2-4d0f-a75c-7827485cb42d","Type":"ContainerStarted","Data":"14a9ff28a6078cfcb41c163e90a0d9ba2abb7f39cf9fe994bc2a0cbc5e5b6cb0"} Nov 03 20:47:16.711528 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:16.711500 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9bt29" event={"ID":"eaf3e4d3-7019-42e5-9e1e-2927b7f22304","Type":"ContainerStarted","Data":"0d1c0d124ccb983ea54849dfaf3549fc40b31296c38641a01591988123f40fc5"} Nov 03 20:47:16.715021 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:16.714994 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" event={"ID":"a2dc043c-52c0-4da0-8370-ed975ebb987f","Type":"ContainerStarted","Data":"546bdb5314f9cefee256c637fee36db01e128465d8deb23c89c9021544635501"} Nov 03 20:47:16.716902 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:16.716879 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-thxwg" event={"ID":"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb","Type":"ContainerStarted","Data":"30c9d23cea06d2e2ee7e26b675a4eadcd7b354fc941609a1dcae96f319a95d40"} Nov 03 20:47:16.774340 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:16.774223 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 20:42:15 +0000 UTC" deadline="2028-08-13 05:21:39.220245294 +0000 UTC" Nov 03 20:47:16.774340 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:16.774282 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15560h34m22.445968576s" Nov 03 20:47:17.300568 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:17.300525 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-metrics-certs\") pod \"network-metrics-daemon-tqhnr\" (UID: \"c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c\") " pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:17.300750 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:17.300711 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:47:17.300829 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:17.300782 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-metrics-certs podName:c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c nodeName:}" failed. No retries permitted until 2026-11-03 20:47:19.300762156 +0000 UTC m=+6.162637582 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-metrics-certs") pod "network-metrics-daemon-tqhnr" (UID: "c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:47:17.401404 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:17.401339 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/712fcecb-4757-4dd0-8f96-084b1e6ee903-original-pull-secret\") pod \"global-pull-secret-syncer-mqmds\" (UID: \"712fcecb-4757-4dd0-8f96-084b1e6ee903\") " pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:17.401646 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:17.401436 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5mgg\" (UniqueName: \"kubernetes.io/projected/955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81-kube-api-access-j5mgg\") pod \"network-check-target-lm85p\" (UID: \"955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81\") " pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:17.401646 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:17.401624 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:47:17.401646 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:17.401647 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:47:17.401850 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:17.401663 2575 projected.go:196] Error preparing data for projected volume kube-api-access-j5mgg for pod openshift-network-diagnostics/network-check-target-lm85p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:47:17.401850 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:17.401727 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81-kube-api-access-j5mgg podName:955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:19.40170764 +0000 UTC m=+6.263583071 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-j5mgg" (UniqueName: "kubernetes.io/projected/955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81-kube-api-access-j5mgg") pod "network-check-target-lm85p" (UID: "955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:47:17.401850 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:17.401834 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:47:17.402019 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:17.401877 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/712fcecb-4757-4dd0-8f96-084b1e6ee903-original-pull-secret podName:712fcecb-4757-4dd0-8f96-084b1e6ee903 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:19.40186364 +0000 UTC m=+6.263739054 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/712fcecb-4757-4dd0-8f96-084b1e6ee903-original-pull-secret") pod "global-pull-secret-syncer-mqmds" (UID: "712fcecb-4757-4dd0-8f96-084b1e6ee903") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:47:17.697454 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:17.697184 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:17.697454 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:17.697227 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:17.697454 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:17.697351 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqhnr" podUID="c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c" Nov 03 20:47:17.697454 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:17.697437 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85p" podUID="955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81" Nov 03 20:47:17.697830 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:17.697499 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:17.697830 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:17.697618 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqmds" podUID="712fcecb-4757-4dd0-8f96-084b1e6ee903" Nov 03 20:47:17.775296 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:17.775186 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 20:42:15 +0000 UTC" deadline="2028-06-21 22:23:44.722740091 +0000 UTC" Nov 03 20:47:17.775296 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:17.775295 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14305h36m26.947450623s" Nov 03 20:47:19.319642 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:19.319599 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-metrics-certs\") pod \"network-metrics-daemon-tqhnr\" (UID: \"c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c\") " pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:19.320104 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:19.319765 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:47:19.320104 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:19.319828 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-metrics-certs podName:c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c nodeName:}" failed. No retries permitted until 2026-11-03 20:47:23.319810586 +0000 UTC m=+10.181686011 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-metrics-certs") pod "network-metrics-daemon-tqhnr" (UID: "c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:47:19.421931 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:19.420522 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/712fcecb-4757-4dd0-8f96-084b1e6ee903-original-pull-secret\") pod \"global-pull-secret-syncer-mqmds\" (UID: \"712fcecb-4757-4dd0-8f96-084b1e6ee903\") " pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:19.421931 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:19.420610 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5mgg\" (UniqueName: \"kubernetes.io/projected/955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81-kube-api-access-j5mgg\") pod \"network-check-target-lm85p\" (UID: \"955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81\") " pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:19.421931 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:19.420754 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:47:19.421931 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:19.420777 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:47:19.421931 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:19.420791 2575 projected.go:196] Error preparing data for projected volume kube-api-access-j5mgg for pod openshift-network-diagnostics/network-check-target-lm85p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:47:19.421931 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:19.420849 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81-kube-api-access-j5mgg podName:955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:23.420830392 +0000 UTC m=+10.282705813 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-j5mgg" (UniqueName: "kubernetes.io/projected/955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81-kube-api-access-j5mgg") pod "network-check-target-lm85p" (UID: "955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:47:19.421931 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:19.420924 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:47:19.421931 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:19.420960 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/712fcecb-4757-4dd0-8f96-084b1e6ee903-original-pull-secret podName:712fcecb-4757-4dd0-8f96-084b1e6ee903 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:23.420946772 +0000 UTC m=+10.282822191 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/712fcecb-4757-4dd0-8f96-084b1e6ee903-original-pull-secret") pod "global-pull-secret-syncer-mqmds" (UID: "712fcecb-4757-4dd0-8f96-084b1e6ee903") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:47:19.695975 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:19.695681 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:19.696153 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:19.696032 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqmds" podUID="712fcecb-4757-4dd0-8f96-084b1e6ee903" Nov 03 20:47:19.696153 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:19.696117 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:19.696264 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:19.696221 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqhnr" podUID="c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c" Nov 03 20:47:19.696264 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:19.696238 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:19.696361 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:19.696346 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85p" podUID="955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81" Nov 03 20:47:20.763957 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:20.763548 2575 generic.go:356] "Generic (PLEG): container finished" podID="55575692b9132c5959a94942a022491a" containerID="a661acc6ec95514f9d77dcd49a98d8a6749e32fbac985a9f2bf140465bfbcfb4" exitCode=0 Nov 03 20:47:20.763957 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:20.763740 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal" event={"ID":"55575692b9132c5959a94942a022491a","Type":"ContainerDied","Data":"a661acc6ec95514f9d77dcd49a98d8a6749e32fbac985a9f2bf140465bfbcfb4"} Nov 03 20:47:20.776179 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:20.776144 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal" event={"ID":"448c85734c0da9850a3328446f82dce9","Type":"ContainerStarted","Data":"50ea737e370d91b3dd5eba1bd2d00c85aeb83f4a79c43b4c678e63b22baf2c58"} Nov 03 20:47:20.789780 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:20.785472 2575 generic.go:356] "Generic (PLEG): container finished" podID="a882e61f-2b83-4ff4-8370-cba401a6d9c8" containerID="85af365816543163ab496b1e4f614b0508143235c68f5486fd9ed1dec0c68d12" exitCode=0 Nov 03 20:47:20.789780 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:20.785541 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5p62l" event={"ID":"a882e61f-2b83-4ff4-8370-cba401a6d9c8","Type":"ContainerDied","Data":"85af365816543163ab496b1e4f614b0508143235c68f5486fd9ed1dec0c68d12"} Nov 03 20:47:21.000513 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:21.000306 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal" podStartSLOduration=6.0002848 podStartE2EDuration="6.0002848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:20.999037873 +0000 UTC m=+7.860913302" watchObservedRunningTime="2026-11-03 20:47:21.0002848 +0000 UTC m=+7.862160230" Nov 03 20:47:21.695235 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:21.695199 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:21.695418 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:21.695317 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqmds" podUID="712fcecb-4757-4dd0-8f96-084b1e6ee903" Nov 03 20:47:21.695418 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:21.695395 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:21.695525 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:21.695465 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqhnr" podUID="c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c" Nov 03 20:47:21.695559 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:21.695535 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:21.695651 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:21.695613 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85p" podUID="955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81" Nov 03 20:47:21.790705 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:21.790666 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ksg72" event={"ID":"cba96b6a-ccc8-4c1c-b371-1bb5ece29afc","Type":"ContainerStarted","Data":"06146a94f4f7f8ac955a60349093db203281d6fa5615ca775df33af9dee2730f"} Nov 03 20:47:21.797406 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:21.797377 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal" event={"ID":"55575692b9132c5959a94942a022491a","Type":"ContainerStarted","Data":"f332c07dc1966a69ec1591df1fc28ff79d4e32a9a21d276aa50baea5ae4eebfc"} Nov 03 20:47:21.910321 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:21.910039 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ksg72" podStartSLOduration=2.794080123 podStartE2EDuration="7.910018039s" totalImagesPullingTime="5.115937916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:21.909130567 +0000 UTC m=+8.771005996" watchObservedRunningTime="2026-11-03 20:47:21.910018039 +0000 UTC m=+8.771893470" Nov 03 20:47:22.807294 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:22.806785 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lmq6t" event={"ID":"bfca9b06-e0d2-4d0f-a75c-7827485cb42d","Type":"ContainerStarted","Data":"ee3edca354ef153897d0b44513dc215369fc9b6480ad2b89eb2b6ec1cc9057d6"} Nov 03 20:47:22.925070 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:22.925007 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal" podStartSLOduration=7.9249864599999995 podStartE2EDuration="7.92498646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:22.02768862 +0000 UTC m=+8.889564052" watchObservedRunningTime="2026-11-03 20:47:22.92498646 +0000 UTC m=+9.786861889" Nov 03 20:47:23.352904 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:23.352864 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-metrics-certs\") pod \"network-metrics-daemon-tqhnr\" (UID: \"c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c\") " pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:23.353104 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:23.353038 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:47:23.353148 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:23.353117 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-metrics-certs podName:c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c nodeName:}" failed. No retries permitted until 2026-11-03 20:47:31.353099112 +0000 UTC m=+18.214974523 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-metrics-certs") pod "network-metrics-daemon-tqhnr" (UID: "c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:47:23.453294 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:23.453254 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5mgg\" (UniqueName: \"kubernetes.io/projected/955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81-kube-api-access-j5mgg\") pod \"network-check-target-lm85p\" (UID: \"955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81\") " pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:23.453477 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:23.453333 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/712fcecb-4757-4dd0-8f96-084b1e6ee903-original-pull-secret\") pod \"global-pull-secret-syncer-mqmds\" (UID: \"712fcecb-4757-4dd0-8f96-084b1e6ee903\") " pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:23.453477 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:23.453469 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:47:23.453587 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:23.453537 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/712fcecb-4757-4dd0-8f96-084b1e6ee903-original-pull-secret podName:712fcecb-4757-4dd0-8f96-084b1e6ee903 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:31.453519608 +0000 UTC m=+18.315395021 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/712fcecb-4757-4dd0-8f96-084b1e6ee903-original-pull-secret") pod "global-pull-secret-syncer-mqmds" (UID: "712fcecb-4757-4dd0-8f96-084b1e6ee903") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:47:23.453657 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:23.453621 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:47:23.453657 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:23.453636 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:47:23.453657 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:23.453648 2575 projected.go:196] Error preparing data for projected volume kube-api-access-j5mgg for pod openshift-network-diagnostics/network-check-target-lm85p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:47:23.453805 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:23.453681 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81-kube-api-access-j5mgg podName:955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:31.453670073 +0000 UTC m=+18.315545482 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-j5mgg" (UniqueName: "kubernetes.io/projected/955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81-kube-api-access-j5mgg") pod "network-check-target-lm85p" (UID: "955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:47:23.694456 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:23.694380 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:23.694456 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:23.694401 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:23.694667 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:23.694380 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:23.694667 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:23.694513 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85p" podUID="955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81" Nov 03 20:47:23.694667 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:23.694632 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqhnr" podUID="c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c" Nov 03 20:47:23.694807 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:23.694704 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqmds" podUID="712fcecb-4757-4dd0-8f96-084b1e6ee903" Nov 03 20:47:23.811917 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:23.811885 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" event={"ID":"a3e654a3-7d76-4793-9627-6334ba6db6d9","Type":"ContainerStarted","Data":"9774c677a38fafa163e0095965a05b0ef2e762c8569e3f3be106df5fed397068"} Nov 03 20:47:24.166218 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:24.164569 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ksg72" Nov 03 20:47:24.166218 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:24.165794 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ksg72" Nov 03 20:47:24.293764 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:24.293696 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lmq6t" podStartSLOduration=4.197474061 podStartE2EDuration="10.293675319s" totalImagesPullingTime="6.096201258s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:47:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:22.925443398 +0000 UTC m=+9.787318829" watchObservedRunningTime="2026-11-03 20:47:24.293675319 +0000 UTC m=+11.155550751" Nov 03 20:47:24.817195 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:24.817132 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-thxwg" event={"ID":"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb","Type":"ContainerStarted","Data":"b9554e0c99311eff198e093d68632360842c6b0e1a72cf72ed148a358b9a4811"} Nov 03 20:47:24.940513 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:24.940448 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-thxwg" podStartSLOduration=2.236999034 podStartE2EDuration="10.940428418s" totalImagesPullingTime="8.703429384s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:47:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:24.93952677 +0000 UTC m=+11.801402202" watchObservedRunningTime="2026-11-03 20:47:24.940428418 +0000 UTC m=+11.802303847" Nov 03 20:47:25.026130 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:25.025882 2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 20:47:25.648808 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:25.648709 2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:47:25.026125809Z","UUID":"01ec36e0-f4d1-4f1c-9330-1453881a7c65","Handler":null,"Name":"","Endpoint":""} Nov 03 20:47:25.650664 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:25.650638 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 20:47:25.650783 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:25.650672 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 20:47:25.695673 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:25.695259 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:25.695673 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:25.695304 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:25.695673 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:25.695412 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85p" podUID="955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81" Nov 03 20:47:25.695673 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:25.695275 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:25.695673 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:25.695543 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqhnr" podUID="c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c" Nov 03 20:47:25.695673 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:25.695620 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqmds" podUID="712fcecb-4757-4dd0-8f96-084b1e6ee903" Nov 03 20:47:25.821784 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:25.821709 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" event={"ID":"a3e654a3-7d76-4793-9627-6334ba6db6d9","Type":"ContainerStarted","Data":"8f8e4e9b1b86871abf229d4f46eff47f6e14f9014a4931960cf9389540894d05"} Nov 03 20:47:25.828835 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:25.828797 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vq2rj" event={"ID":"9050192a-0ec2-45b9-8b2c-2beb0501b09b","Type":"ContainerStarted","Data":"1b7a2e878382f3ad651bfbcd793583d4fa2b15cc85027d6f11e974a2612af0e9"} Nov 03 20:47:25.831396 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:25.831367 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5p62l" event={"ID":"a882e61f-2b83-4ff4-8370-cba401a6d9c8","Type":"ContainerStarted","Data":"7c3a4eb1e5feb2f6c8f7a0139cedc0cae97ecd31d38d35e6fd76f1e0bd052e31"} Nov 03 20:47:25.947861 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:25.947805 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-vq2rj" podStartSLOduration=2.916976059 podStartE2EDuration="11.947792867s" totalImagesPullingTime="9.030816808s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:47:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:25.947144071 +0000 UTC m=+12.809019501" watchObservedRunningTime="2026-11-03 20:47:25.947792867 +0000 UTC m=+12.809668295" Nov 03 20:47:26.835625 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:26.835590 2575 generic.go:356] "Generic (PLEG): container finished" podID="a882e61f-2b83-4ff4-8370-cba401a6d9c8" containerID="7c3a4eb1e5feb2f6c8f7a0139cedc0cae97ecd31d38d35e6fd76f1e0bd052e31" exitCode=0 Nov 03 20:47:26.836155 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:26.835663 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5p62l" event={"ID":"a882e61f-2b83-4ff4-8370-cba401a6d9c8","Type":"ContainerDied","Data":"7c3a4eb1e5feb2f6c8f7a0139cedc0cae97ecd31d38d35e6fd76f1e0bd052e31"} Nov 03 20:47:26.837756 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:26.837729 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9bt29" event={"ID":"eaf3e4d3-7019-42e5-9e1e-2927b7f22304","Type":"ContainerStarted","Data":"dd6c215d0ad7b3ebb739cad286d6347dab55a877c3ab4ea471bfd690e05baa5b"} Nov 03 20:47:27.080883 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:27.080663 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9bt29" podStartSLOduration=3.838933787 podStartE2EDuration="13.080641107s" totalImagesPullingTime="9.24170732s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:47:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:27.080427264 +0000 UTC m=+13.942302691" watchObservedRunningTime="2026-11-03 20:47:27.080641107 +0000 UTC m=+13.942516536" Nov 03 20:47:27.693149 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:27.693111 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:27.693332 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:27.693154 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:27.693332 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:27.693256 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqmds" podUID="712fcecb-4757-4dd0-8f96-084b1e6ee903" Nov 03 20:47:27.693332 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:27.693309 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqhnr" podUID="c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c" Nov 03 20:47:27.693514 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:27.693320 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:27.693514 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:27.693398 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85p" podUID="955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81" Nov 03 20:47:27.841931 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:27.841893 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" event={"ID":"026b5dc8-bcff-4afc-a784-baf95c1cec27","Type":"ContainerStarted","Data":"5625332fef3dda5f23e31ca47c3857735d4aa98aebbaa50ba552f6d88a055d63"} Nov 03 20:47:27.844528 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:27.844497 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" event={"ID":"a3e654a3-7d76-4793-9627-6334ba6db6d9","Type":"ContainerStarted","Data":"7fa7d098935c19dcd66e816abd82d38854b2b6896f59bad09bef03c4b55be926"} Nov 03 20:47:28.084713 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:28.084653 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" podStartSLOduration=3.120834769 podStartE2EDuration="14.084634819s" totalImagesPullingTime="10.96380005s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:47:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:27.963027629 +0000 UTC m=+14.824903057" watchObservedRunningTime="2026-11-03 20:47:28.084634819 +0000 UTC m=+14.946510250" Nov 03 20:47:28.085345 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:28.085311 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" podStartSLOduration=3.51865655 podStartE2EDuration="14.08530292s" totalImagesPullingTime="10.56664637s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:47:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:28.084451665 +0000 UTC m=+14.946327095" watchObservedRunningTime="2026-11-03 20:47:28.08530292 +0000 UTC m=+14.947178352" Nov 03 20:47:28.848536 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:28.848457 2575 generic.go:356] "Generic (PLEG): container finished" podID="a882e61f-2b83-4ff4-8370-cba401a6d9c8" containerID="567443ba70810871551abbc60ed13abaae9134401d1d69cb78e333afec118431" exitCode=0 Nov 03 20:47:28.849124 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:28.848546 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5p62l" event={"ID":"a882e61f-2b83-4ff4-8370-cba401a6d9c8","Type":"ContainerDied","Data":"567443ba70810871551abbc60ed13abaae9134401d1d69cb78e333afec118431"} Nov 03 20:47:29.693129 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:29.692908 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:29.693294 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:29.692908 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:29.693294 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:29.693228 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqhnr" podUID="c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c" Nov 03 20:47:29.693294 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:29.692927 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:29.693467 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:29.693320 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqmds" podUID="712fcecb-4757-4dd0-8f96-084b1e6ee903" Nov 03 20:47:29.693467 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:29.693364 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85p" podUID="955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81" Nov 03 20:47:30.856485 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:30.856437 2575 generic.go:356] "Generic (PLEG): container finished" podID="a882e61f-2b83-4ff4-8370-cba401a6d9c8" containerID="29dcb550d7251ec444ded1fe2ef043332cb33107a2df27b4f438d10d583800a5" exitCode=0 Nov 03 20:47:30.856979 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:30.856500 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5p62l" event={"ID":"a882e61f-2b83-4ff4-8370-cba401a6d9c8","Type":"ContainerDied","Data":"29dcb550d7251ec444ded1fe2ef043332cb33107a2df27b4f438d10d583800a5"} Nov 03 20:47:31.420061 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:31.420026 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-metrics-certs\") pod \"network-metrics-daemon-tqhnr\" (UID: \"c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c\") " pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:31.420233 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:31.420183 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:47:31.420273 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:31.420248 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-metrics-certs podName:c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c nodeName:}" failed. No retries permitted until 2026-11-03 20:47:47.420232674 +0000 UTC m=+34.282108084 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-metrics-certs") pod "network-metrics-daemon-tqhnr" (UID: "c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:47:31.521250 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:31.521212 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5mgg\" (UniqueName: \"kubernetes.io/projected/955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81-kube-api-access-j5mgg\") pod \"network-check-target-lm85p\" (UID: \"955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81\") " pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:31.521416 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:31.521274 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/712fcecb-4757-4dd0-8f96-084b1e6ee903-original-pull-secret\") pod \"global-pull-secret-syncer-mqmds\" (UID: \"712fcecb-4757-4dd0-8f96-084b1e6ee903\") " pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:31.521416 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:31.521382 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:47:31.521416 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:31.521386 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:47:31.521416 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:31.521405 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:47:31.521545 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:31.521418 2575 projected.go:196] Error preparing data for projected volume kube-api-access-j5mgg for pod openshift-network-diagnostics/network-check-target-lm85p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:47:31.521545 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:31.521449 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/712fcecb-4757-4dd0-8f96-084b1e6ee903-original-pull-secret podName:712fcecb-4757-4dd0-8f96-084b1e6ee903 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:47.521429883 +0000 UTC m=+34.383305307 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/712fcecb-4757-4dd0-8f96-084b1e6ee903-original-pull-secret") pod "global-pull-secret-syncer-mqmds" (UID: "712fcecb-4757-4dd0-8f96-084b1e6ee903") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:47:31.521545 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:31.521463 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81-kube-api-access-j5mgg podName:955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:47.521457331 +0000 UTC m=+34.383332737 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-j5mgg" (UniqueName: "kubernetes.io/projected/955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81-kube-api-access-j5mgg") pod "network-check-target-lm85p" (UID: "955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:47:31.692730 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:31.692644 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:31.692730 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:31.692687 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:31.692918 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:31.692760 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqmds" podUID="712fcecb-4757-4dd0-8f96-084b1e6ee903" Nov 03 20:47:31.692918 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:31.692828 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqhnr" podUID="c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c" Nov 03 20:47:31.692918 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:31.692862 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:31.693014 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:31.692948 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85p" podUID="955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81" Nov 03 20:47:31.697254 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:31.697224 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ksg72_cba96b6a-ccc8-4c1c-b371-1bb5ece29afc/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:32.375173 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:32.375144 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vq2rj_9050192a-0ec2-45b9-8b2c-2beb0501b09b/dns-node-resolver/0.log" Nov 03 20:47:32.863577 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:32.863546 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" event={"ID":"a2dc043c-52c0-4da0-8370-ed975ebb987f","Type":"ContainerStarted","Data":"7ecd9d8ecd36984947476e9f83edb54df24444ce9cb68b1cb633132fc12754b7"} Nov 03 20:47:32.863727 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:32.863634 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" event={"ID":"a2dc043c-52c0-4da0-8370-ed975ebb987f","Type":"ContainerStarted","Data":"409a39b81bdd59a081d1dbcaff79a41550b56a0eee30c73f9b1ad6b404322d7d"} Nov 03 20:47:32.863727 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:32.863685 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" event={"ID":"a2dc043c-52c0-4da0-8370-ed975ebb987f","Type":"ContainerStarted","Data":"46db8ce4128b25ec7a15110a7d9011326f9c517cd7d7b7b818387a0c9e4fbabf"} Nov 03 20:47:32.863727 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:32.863717 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" event={"ID":"a2dc043c-52c0-4da0-8370-ed975ebb987f","Type":"ContainerStarted","Data":"2c8a253313f65be6ceadc7d01f4ea0d18977f7a4efe99ef93163bc6d6dcb3073"} Nov 03 20:47:32.863825 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:32.863743 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" event={"ID":"a2dc043c-52c0-4da0-8370-ed975ebb987f","Type":"ContainerStarted","Data":"c31108d58726647a214043b69fda7d882f456e4fffad58709f4a1dc7fee24db2"} Nov 03 20:47:32.863825 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:32.863771 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" event={"ID":"a2dc043c-52c0-4da0-8370-ed975ebb987f","Type":"ContainerStarted","Data":"bbbc442ce3e3ad432a0b44e8bc53ce81c82225701a81997fe8d5419db9cda019"} Nov 03 20:47:32.974917 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:32.974891 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lmq6t_bfca9b06-e0d2-4d0f-a75c-7827485cb42d/node-ca/0.log" Nov 03 20:47:33.693798 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:33.693760 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:33.694512 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:33.693883 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:33.694512 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:33.693904 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqmds" podUID="712fcecb-4757-4dd0-8f96-084b1e6ee903" Nov 03 20:47:33.694512 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:33.693944 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqhnr" podUID="c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c" Nov 03 20:47:33.694512 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:33.693991 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:33.694512 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:33.694089 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85p" podUID="955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81" Nov 03 20:47:33.868446 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:33.868415 2575 generic.go:356] "Generic (PLEG): container finished" podID="a882e61f-2b83-4ff4-8370-cba401a6d9c8" containerID="d54c5c98b47d88b4fbfe5ad565ae0c278bd92030464c7a77fa8c1c1d8e1ad515" exitCode=0 Nov 03 20:47:33.868581 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:33.868463 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5p62l" event={"ID":"a882e61f-2b83-4ff4-8370-cba401a6d9c8","Type":"ContainerDied","Data":"d54c5c98b47d88b4fbfe5ad565ae0c278bd92030464c7a77fa8c1c1d8e1ad515"} Nov 03 20:47:34.873778 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:34.873744 2575 generic.go:356] "Generic (PLEG): container finished" podID="a882e61f-2b83-4ff4-8370-cba401a6d9c8" containerID="9365311a852bade283bae3aa2130a09998d0e2075ec725e0691a0c872992072d" exitCode=0 Nov 03 20:47:34.874274 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:34.873822 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5p62l" event={"ID":"a882e61f-2b83-4ff4-8370-cba401a6d9c8","Type":"ContainerDied","Data":"9365311a852bade283bae3aa2130a09998d0e2075ec725e0691a0c872992072d"} Nov 03 20:47:34.877033 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:34.877013 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" event={"ID":"a2dc043c-52c0-4da0-8370-ed975ebb987f","Type":"ContainerStarted","Data":"904fded4aa09555f430be2c45b00a3b6cd6ed5548011a005f423210a9a9e3f9d"} Nov 03 20:47:35.693203 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:35.693168 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:35.693203 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:35.693190 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:35.693203 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:35.693204 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:35.693523 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:35.693296 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqmds" podUID="712fcecb-4757-4dd0-8f96-084b1e6ee903" Nov 03 20:47:35.693523 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:35.693457 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqhnr" podUID="c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c" Nov 03 20:47:35.693618 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:35.693564 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85p" podUID="955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81" Nov 03 20:47:35.882873 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:35.882838 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5p62l" event={"ID":"a882e61f-2b83-4ff4-8370-cba401a6d9c8","Type":"ContainerStarted","Data":"afbc650d4a39773b5dad40de304fed2a0d298d6507b6df5398ebc46b321fddd3"} Nov 03 20:47:36.010497 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:36.010446 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5p62l" podStartSLOduration=8.412353323 podStartE2EDuration="22.010433188s" totalImagesPullingTime="13.598079865s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:47:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:36.010350333 +0000 UTC m=+22.872225762" watchObservedRunningTime="2026-11-03 20:47:36.010433188 +0000 UTC m=+22.872308615" Nov 03 20:47:36.887998 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:36.887957 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" event={"ID":"a2dc043c-52c0-4da0-8370-ed975ebb987f","Type":"ContainerStarted","Data":"f5c92e026fb551bf0b55bb94dbcf84150132d23cf086469579596f55e355877d"} Nov 03 20:47:37.071010 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:37.070951 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" podStartSLOduration=7.24836522 podStartE2EDuration="23.070938406s" totalImagesPullingTime="15.822573186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:47:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:37.070707729 +0000 UTC m=+23.932583157" watchObservedRunningTime="2026-11-03 20:47:37.070938406 +0000 UTC m=+23.932813834" Nov 03 20:47:37.692961 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:37.692924 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:37.693194 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:37.692934 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:37.693194 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:37.693037 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqhnr" podUID="c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c" Nov 03 20:47:37.693194 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:37.692937 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:37.693194 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:37.693117 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85p" podUID="955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81" Nov 03 20:47:37.693194 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:37.693191 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqmds" podUID="712fcecb-4757-4dd0-8f96-084b1e6ee903" Nov 03 20:47:37.890666 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:37.890635 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:47:37.891096 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:37.891044 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:37.891164 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:37.891152 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:37.907256 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:37.907225 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:37.907420 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:37.907361 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:38.892620 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:38.892592 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:47:39.693453 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:39.693419 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:39.693628 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:39.693528 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:39.693628 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:39.693539 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85p" podUID="955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81" Nov 03 20:47:39.693704 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:39.693630 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqhnr" podUID="c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c" Nov 03 20:47:39.693704 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:39.693679 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:39.693763 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:39.693734 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqmds" podUID="712fcecb-4757-4dd0-8f96-084b1e6ee903" Nov 03 20:47:39.894916 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:39.894887 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:47:41.046595 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:41.046559 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:47:41.047162 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:41.046817 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:47:41.057029 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:41.056990 2575 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" podUID="a2dc043c-52c0-4da0-8370-ed975ebb987f" containerName="ovnkube-controller" probeResult="failure" output="" Nov 03 20:47:41.065624 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:41.065589 2575 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" podUID="a2dc043c-52c0-4da0-8370-ed975ebb987f" containerName="ovnkube-controller" probeResult="failure" output="" Nov 03 20:47:41.693275 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:41.693251 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:41.693449 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:41.693298 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:41.693449 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:41.693309 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:41.693449 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:41.693390 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqmds" podUID="712fcecb-4757-4dd0-8f96-084b1e6ee903" Nov 03 20:47:41.693449 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:41.693429 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqhnr" podUID="c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c" Nov 03 20:47:41.693673 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:41.693530 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85p" podUID="955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81" Nov 03 20:47:43.694924 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:43.694891 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:43.695340 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:43.694977 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:43.695340 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:43.695008 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:43.695340 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:43.695022 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqmds" podUID="712fcecb-4757-4dd0-8f96-084b1e6ee903" Nov 03 20:47:43.695340 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:43.695099 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85p" podUID="955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81" Nov 03 20:47:43.695340 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:43.695149 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqhnr" podUID="c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c" Nov 03 20:47:45.497437 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:45.497396 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ksg72" Nov 03 20:47:45.497896 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:45.497586 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:47:45.498009 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:45.497984 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ksg72" Nov 03 20:47:45.693689 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:45.693647 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:45.693880 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:45.693647 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:45.693880 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:45.693765 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:45.693880 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:45.693777 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqmds" podUID="712fcecb-4757-4dd0-8f96-084b1e6ee903" Nov 03 20:47:45.693880 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:45.693854 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85p" podUID="955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81" Nov 03 20:47:45.694053 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:45.693948 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqhnr" podUID="c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c" Nov 03 20:47:47.434250 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:47.434218 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-metrics-certs\") pod \"network-metrics-daemon-tqhnr\" (UID: \"c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c\") " pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:47.434633 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:47.434334 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:47:47.434633 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:47.434385 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-metrics-certs podName:c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c nodeName:}" failed. No retries permitted until 2026-11-03 20:48:19.434372925 +0000 UTC m=+66.296248345 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-metrics-certs") pod "network-metrics-daemon-tqhnr" (UID: "c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:47:47.534819 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:47.534783 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5mgg\" (UniqueName: \"kubernetes.io/projected/955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81-kube-api-access-j5mgg\") pod \"network-check-target-lm85p\" (UID: \"955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81\") " pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:47.535010 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:47.534831 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/712fcecb-4757-4dd0-8f96-084b1e6ee903-original-pull-secret\") pod \"global-pull-secret-syncer-mqmds\" (UID: \"712fcecb-4757-4dd0-8f96-084b1e6ee903\") " pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:47.535010 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:47.534934 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:47:47.535010 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:47.534960 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:47:47.535010 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:47.534980 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/712fcecb-4757-4dd0-8f96-084b1e6ee903-original-pull-secret podName:712fcecb-4757-4dd0-8f96-084b1e6ee903 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:19.534968644 +0000 UTC m=+66.396844050 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/712fcecb-4757-4dd0-8f96-084b1e6ee903-original-pull-secret") pod "global-pull-secret-syncer-mqmds" (UID: "712fcecb-4757-4dd0-8f96-084b1e6ee903") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:47:47.535010 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:47.534983 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:47:47.535010 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:47.534996 2575 projected.go:196] Error preparing data for projected volume kube-api-access-j5mgg for pod openshift-network-diagnostics/network-check-target-lm85p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:47:47.535261 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:47.535043 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81-kube-api-access-j5mgg podName:955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:19.535027875 +0000 UTC m=+66.396903297 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-j5mgg" (UniqueName: "kubernetes.io/projected/955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81-kube-api-access-j5mgg") pod "network-check-target-lm85p" (UID: "955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:47:47.693322 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:47.693237 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:47.693322 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:47.693263 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:47.693322 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:47.693237 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:47.693600 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:47.693374 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqhnr" podUID="c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c" Nov 03 20:47:47.693600 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:47.693473 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85p" podUID="955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81" Nov 03 20:47:47.693600 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:47.693546 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqmds" podUID="712fcecb-4757-4dd0-8f96-084b1e6ee903" Nov 03 20:47:49.693787 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:49.693753 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:49.694223 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:49.693824 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:49.694223 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:49.693821 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:49.694223 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:49.693922 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqhnr" podUID="c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c" Nov 03 20:47:49.694223 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:49.693987 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85p" podUID="955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81" Nov 03 20:47:49.694223 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:49.694097 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqmds" podUID="712fcecb-4757-4dd0-8f96-084b1e6ee903" Nov 03 20:47:51.696033 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:51.695998 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:51.696612 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:51.696012 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:51.696612 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:51.696148 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85p" podUID="955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81" Nov 03 20:47:51.696612 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:51.696165 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:51.696612 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:51.696268 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqhnr" podUID="c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c" Nov 03 20:47:51.696612 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:51.696348 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqmds" podUID="712fcecb-4757-4dd0-8f96-084b1e6ee903" Nov 03 20:47:53.119471 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:53.119226 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mqmds"] Nov 03 20:47:53.119854 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:53.119621 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:53.119854 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:53.119743 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqmds" podUID="712fcecb-4757-4dd0-8f96-084b1e6ee903" Nov 03 20:47:53.121858 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:53.121834 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lm85p"] Nov 03 20:47:53.121985 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:53.121925 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:53.122051 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:53.121995 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85p" podUID="955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81" Nov 03 20:47:53.122435 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:53.122414 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tqhnr"] Nov 03 20:47:53.122538 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:53.122526 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:53.122646 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:53.122628 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqhnr" podUID="c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c" Nov 03 20:47:54.692988 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:54.692949 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:54.693504 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:54.693008 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:54.693504 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:54.693114 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:54.693504 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:54.693129 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqhnr" podUID="c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c" Nov 03 20:47:54.693504 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:54.693187 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqmds" podUID="712fcecb-4757-4dd0-8f96-084b1e6ee903" Nov 03 20:47:54.693504 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:54.693244 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85p" podUID="955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81" Nov 03 20:47:56.693164 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:56.693059 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:56.693675 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:56.693059 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:56.693675 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:56.693218 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqhnr" podUID="c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c" Nov 03 20:47:56.693675 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:56.693059 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:56.693675 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:56.693254 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqmds" podUID="712fcecb-4757-4dd0-8f96-084b1e6ee903" Nov 03 20:47:56.693675 ip-10-0-129-192 kubenswrapper[2575]: E1103 20:47:56.693368 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85p" podUID="955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81" Nov 03 20:47:58.332595 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.332570 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeReady" Nov 03 20:47:58.333056 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.332709 2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 20:47:58.452089 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.452045 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7cvm2"] Nov 03 20:47:58.472953 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.472926 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7cvm2" Nov 03 20:47:58.503904 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.503874 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tms4q"] Nov 03 20:47:58.509256 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.509218 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7vv52"] Nov 03 20:47:58.509414 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.509348 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tms4q" Nov 03 20:47:58.517600 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.517573 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7vv52" Nov 03 20:47:58.527608 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.527586 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 20:47:58.527717 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.527615 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 20:47:58.527717 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.527637 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-68sfv\"" Nov 03 20:47:58.552707 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.552635 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7vv52"] Nov 03 20:47:58.555631 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.555604 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tms4q"] Nov 03 20:47:58.556137 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.556115 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7cvm2"] Nov 03 20:47:58.612550 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.612516 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e64dc44d-5470-4abb-8155-fdf7f24c253b-crio-socket\") pod \"insights-runtime-extractor-tms4q\" (UID: \"e64dc44d-5470-4abb-8155-fdf7f24c253b\") " pod="openshift-insights/insights-runtime-extractor-tms4q" Nov 03 20:47:58.612550 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.612554 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e64dc44d-5470-4abb-8155-fdf7f24c253b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tms4q\" (UID: \"e64dc44d-5470-4abb-8155-fdf7f24c253b\") " pod="openshift-insights/insights-runtime-extractor-tms4q" Nov 03 20:47:58.612797 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.612573 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k85jh\" (UniqueName: \"kubernetes.io/projected/fac5da49-6649-4779-9529-089804436313-kube-api-access-k85jh\") pod \"ingress-canary-7vv52\" (UID: \"fac5da49-6649-4779-9529-089804436313\") " pod="openshift-ingress-canary/ingress-canary-7vv52" Nov 03 20:47:58.612797 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.612696 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/05683773-8c40-4d7c-9cc9-ee9d71b25b23-config-volume\") pod \"dns-default-7cvm2\" (UID: \"05683773-8c40-4d7c-9cc9-ee9d71b25b23\") " pod="openshift-dns/dns-default-7cvm2" Nov 03 20:47:58.612797 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.612731 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05683773-8c40-4d7c-9cc9-ee9d71b25b23-metrics-tls\") pod \"dns-default-7cvm2\" (UID: \"05683773-8c40-4d7c-9cc9-ee9d71b25b23\") " pod="openshift-dns/dns-default-7cvm2" Nov 03 20:47:58.612797 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.612761 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc6n7\" (UniqueName: \"kubernetes.io/projected/05683773-8c40-4d7c-9cc9-ee9d71b25b23-kube-api-access-wc6n7\") pod \"dns-default-7cvm2\" (UID: \"05683773-8c40-4d7c-9cc9-ee9d71b25b23\") " pod="openshift-dns/dns-default-7cvm2" Nov 03 20:47:58.612925 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.612831 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05683773-8c40-4d7c-9cc9-ee9d71b25b23-tmp-dir\") pod \"dns-default-7cvm2\" (UID: \"05683773-8c40-4d7c-9cc9-ee9d71b25b23\") " pod="openshift-dns/dns-default-7cvm2" Nov 03 20:47:58.612925 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.612857 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzx44\" (UniqueName: \"kubernetes.io/projected/e64dc44d-5470-4abb-8155-fdf7f24c253b-kube-api-access-zzx44\") pod \"insights-runtime-extractor-tms4q\" (UID: \"e64dc44d-5470-4abb-8155-fdf7f24c253b\") " pod="openshift-insights/insights-runtime-extractor-tms4q" Nov 03 20:47:58.612925 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.612872 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fac5da49-6649-4779-9529-089804436313-cert\") pod \"ingress-canary-7vv52\" (UID: \"fac5da49-6649-4779-9529-089804436313\") " pod="openshift-ingress-canary/ingress-canary-7vv52" Nov 03 20:47:58.613009 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.612935 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e64dc44d-5470-4abb-8155-fdf7f24c253b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tms4q\" (UID: \"e64dc44d-5470-4abb-8155-fdf7f24c253b\") " pod="openshift-insights/insights-runtime-extractor-tms4q" Nov 03 20:47:58.613009 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.612983 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e64dc44d-5470-4abb-8155-fdf7f24c253b-data-volume\") pod \"insights-runtime-extractor-tms4q\" (UID: \"e64dc44d-5470-4abb-8155-fdf7f24c253b\") " pod="openshift-insights/insights-runtime-extractor-tms4q" Nov 03 20:47:58.628504 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.628440 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 20:47:58.628504 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.628464 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 20:47:58.628692 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.628531 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 20:47:58.628692 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.628553 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 20:47:58.628692 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.628558 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 20:47:58.628692 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.628559 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mxhpl\"" Nov 03 20:47:58.628692 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.628609 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 20:47:58.628692 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.628615 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 20:47:58.628692 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.628625 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-d7jn9\"" Nov 03 20:47:58.693261 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.693230 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:47:58.693261 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.693261 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:47:58.693475 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.693459 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:47:58.713432 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.713411 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e64dc44d-5470-4abb-8155-fdf7f24c253b-crio-socket\") pod \"insights-runtime-extractor-tms4q\" (UID: \"e64dc44d-5470-4abb-8155-fdf7f24c253b\") " pod="openshift-insights/insights-runtime-extractor-tms4q" Nov 03 20:47:58.713549 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.713442 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e64dc44d-5470-4abb-8155-fdf7f24c253b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tms4q\" (UID: \"e64dc44d-5470-4abb-8155-fdf7f24c253b\") " pod="openshift-insights/insights-runtime-extractor-tms4q" Nov 03 20:47:58.713549 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.713465 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k85jh\" (UniqueName: \"kubernetes.io/projected/fac5da49-6649-4779-9529-089804436313-kube-api-access-k85jh\") pod \"ingress-canary-7vv52\" (UID: \"fac5da49-6649-4779-9529-089804436313\") " pod="openshift-ingress-canary/ingress-canary-7vv52" Nov 03 20:47:58.713549 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.713533 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/05683773-8c40-4d7c-9cc9-ee9d71b25b23-config-volume\") pod \"dns-default-7cvm2\" (UID: \"05683773-8c40-4d7c-9cc9-ee9d71b25b23\") " pod="openshift-dns/dns-default-7cvm2" Nov 03 20:47:58.713696 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.713612 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e64dc44d-5470-4abb-8155-fdf7f24c253b-crio-socket\") pod \"insights-runtime-extractor-tms4q\" (UID: \"e64dc44d-5470-4abb-8155-fdf7f24c253b\") " pod="openshift-insights/insights-runtime-extractor-tms4q" Nov 03 20:47:58.713696 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.713662 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05683773-8c40-4d7c-9cc9-ee9d71b25b23-metrics-tls\") pod \"dns-default-7cvm2\" (UID: \"05683773-8c40-4d7c-9cc9-ee9d71b25b23\") " pod="openshift-dns/dns-default-7cvm2" Nov 03 20:47:58.713696 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.713688 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc6n7\" (UniqueName: \"kubernetes.io/projected/05683773-8c40-4d7c-9cc9-ee9d71b25b23-kube-api-access-wc6n7\") pod \"dns-default-7cvm2\" (UID: \"05683773-8c40-4d7c-9cc9-ee9d71b25b23\") " pod="openshift-dns/dns-default-7cvm2" Nov 03 20:47:58.713841 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.713732 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05683773-8c40-4d7c-9cc9-ee9d71b25b23-tmp-dir\") pod \"dns-default-7cvm2\" (UID: \"05683773-8c40-4d7c-9cc9-ee9d71b25b23\") " pod="openshift-dns/dns-default-7cvm2" Nov 03 20:47:58.713958 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.713937 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzx44\" (UniqueName: \"kubernetes.io/projected/e64dc44d-5470-4abb-8155-fdf7f24c253b-kube-api-access-zzx44\") pod \"insights-runtime-extractor-tms4q\" (UID: \"e64dc44d-5470-4abb-8155-fdf7f24c253b\") " pod="openshift-insights/insights-runtime-extractor-tms4q" Nov 03 20:47:58.714014 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.713974 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fac5da49-6649-4779-9529-089804436313-cert\") pod \"ingress-canary-7vv52\" (UID: \"fac5da49-6649-4779-9529-089804436313\") " pod="openshift-ingress-canary/ingress-canary-7vv52" Nov 03 20:47:58.714065 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.714018 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/05683773-8c40-4d7c-9cc9-ee9d71b25b23-config-volume\") pod \"dns-default-7cvm2\" (UID: \"05683773-8c40-4d7c-9cc9-ee9d71b25b23\") " pod="openshift-dns/dns-default-7cvm2" Nov 03 20:47:58.714065 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.714020 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05683773-8c40-4d7c-9cc9-ee9d71b25b23-tmp-dir\") pod \"dns-default-7cvm2\" (UID: \"05683773-8c40-4d7c-9cc9-ee9d71b25b23\") " pod="openshift-dns/dns-default-7cvm2" Nov 03 20:47:58.714065 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.714031 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e64dc44d-5470-4abb-8155-fdf7f24c253b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tms4q\" (UID: \"e64dc44d-5470-4abb-8155-fdf7f24c253b\") " pod="openshift-insights/insights-runtime-extractor-tms4q" Nov 03 20:47:58.714211 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.714184 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e64dc44d-5470-4abb-8155-fdf7f24c253b-data-volume\") pod \"insights-runtime-extractor-tms4q\" (UID: \"e64dc44d-5470-4abb-8155-fdf7f24c253b\") " pod="openshift-insights/insights-runtime-extractor-tms4q" Nov 03 20:47:58.714468 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.714450 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e64dc44d-5470-4abb-8155-fdf7f24c253b-data-volume\") pod \"insights-runtime-extractor-tms4q\" (UID: \"e64dc44d-5470-4abb-8155-fdf7f24c253b\") " pod="openshift-insights/insights-runtime-extractor-tms4q" Nov 03 20:47:58.714565 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.714548 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e64dc44d-5470-4abb-8155-fdf7f24c253b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tms4q\" (UID: \"e64dc44d-5470-4abb-8155-fdf7f24c253b\") " pod="openshift-insights/insights-runtime-extractor-tms4q" Nov 03 20:47:58.717528 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.717502 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05683773-8c40-4d7c-9cc9-ee9d71b25b23-metrics-tls\") pod \"dns-default-7cvm2\" (UID: \"05683773-8c40-4d7c-9cc9-ee9d71b25b23\") " pod="openshift-dns/dns-default-7cvm2" Nov 03 20:47:58.717604 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.717594 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fac5da49-6649-4779-9529-089804436313-cert\") pod \"ingress-canary-7vv52\" (UID: \"fac5da49-6649-4779-9529-089804436313\") " pod="openshift-ingress-canary/ingress-canary-7vv52" Nov 03 20:47:58.717642 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.717592 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e64dc44d-5470-4abb-8155-fdf7f24c253b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tms4q\" (UID: \"e64dc44d-5470-4abb-8155-fdf7f24c253b\") " pod="openshift-insights/insights-runtime-extractor-tms4q" Nov 03 20:47:58.748034 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.748002 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 20:47:58.748229 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.748188 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 20:47:58.748394 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.748375 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-snm99\"" Nov 03 20:47:58.748443 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.748426 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-78pt4\"" Nov 03 20:47:58.748494 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.748447 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l9gsz\"" Nov 03 20:47:58.748494 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.748461 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 20:47:58.748494 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.748461 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 20:47:58.777094 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.777044 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzx44\" (UniqueName: \"kubernetes.io/projected/e64dc44d-5470-4abb-8155-fdf7f24c253b-kube-api-access-zzx44\") pod \"insights-runtime-extractor-tms4q\" (UID: \"e64dc44d-5470-4abb-8155-fdf7f24c253b\") " pod="openshift-insights/insights-runtime-extractor-tms4q" Nov 03 20:47:58.777213 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.777156 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc6n7\" (UniqueName: \"kubernetes.io/projected/05683773-8c40-4d7c-9cc9-ee9d71b25b23-kube-api-access-wc6n7\") pod \"dns-default-7cvm2\" (UID: \"05683773-8c40-4d7c-9cc9-ee9d71b25b23\") " pod="openshift-dns/dns-default-7cvm2" Nov 03 20:47:58.777253 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.777241 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k85jh\" (UniqueName: \"kubernetes.io/projected/fac5da49-6649-4779-9529-089804436313-kube-api-access-k85jh\") pod \"ingress-canary-7vv52\" (UID: \"fac5da49-6649-4779-9529-089804436313\") " pod="openshift-ingress-canary/ingress-canary-7vv52" Nov 03 20:47:58.793936 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.793913 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7cvm2" Nov 03 20:47:58.842592 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.842552 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tms4q" Nov 03 20:47:58.847749 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:58.847717 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7vv52" Nov 03 20:47:59.229402 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:59.229283 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7cvm2"] Nov 03 20:47:59.241712 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:59.240368 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfac5da49_6649_4779_9529_089804436313.slice/crio-eed08506c695a4b7e75761bc37391ce9891831f7f93b47b1d781fd9c62ef6d8f WatchSource:0}: Error finding container eed08506c695a4b7e75761bc37391ce9891831f7f93b47b1d781fd9c62ef6d8f: Status 404 returned error can't find the container with id eed08506c695a4b7e75761bc37391ce9891831f7f93b47b1d781fd9c62ef6d8f Nov 03 20:47:59.242384 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:59.242360 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7vv52"] Nov 03 20:47:59.245334 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:59.245316 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tms4q"] Nov 03 20:47:59.245814 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:47:59.245791 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode64dc44d_5470_4abb_8155_fdf7f24c253b.slice/crio-8fb0cf8cc04cb3dc87b3cab3765ae19afff3daf13b90b98b1286baca880ba23b WatchSource:0}: Error finding container 8fb0cf8cc04cb3dc87b3cab3765ae19afff3daf13b90b98b1286baca880ba23b: Status 404 returned error can't find the container with id 8fb0cf8cc04cb3dc87b3cab3765ae19afff3daf13b90b98b1286baca880ba23b Nov 03 20:47:59.938053 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:59.938015 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7vv52" event={"ID":"fac5da49-6649-4779-9529-089804436313","Type":"ContainerStarted","Data":"eed08506c695a4b7e75761bc37391ce9891831f7f93b47b1d781fd9c62ef6d8f"} Nov 03 20:47:59.939408 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:59.939388 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tms4q" event={"ID":"e64dc44d-5470-4abb-8155-fdf7f24c253b","Type":"ContainerStarted","Data":"196ab8853626713208b87c81fc056a261693af5a0e379cf4fc4d83d37e201a96"} Nov 03 20:47:59.939507 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:59.939430 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tms4q" event={"ID":"e64dc44d-5470-4abb-8155-fdf7f24c253b","Type":"ContainerStarted","Data":"8fb0cf8cc04cb3dc87b3cab3765ae19afff3daf13b90b98b1286baca880ba23b"} Nov 03 20:47:59.940371 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:47:59.940352 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7cvm2" event={"ID":"05683773-8c40-4d7c-9cc9-ee9d71b25b23","Type":"ContainerStarted","Data":"0308d0c818cb3fa16f8a663af38380509a71f2b74e894937f2e4961830a36dc2"} Nov 03 20:48:01.948389 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:01.948312 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tms4q" event={"ID":"e64dc44d-5470-4abb-8155-fdf7f24c253b","Type":"ContainerStarted","Data":"abc58467a1af50c840db37a0f73a8ca8f4c1f003e158bdf7ecf2ff2f20376e86"} Nov 03 20:48:02.953136 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:02.952900 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7vv52" event={"ID":"fac5da49-6649-4779-9529-089804436313","Type":"ContainerStarted","Data":"91bd7f82964f2c1a33030abc5ff594b110dd2f48bb86a3317886a24720944900"} Nov 03 20:48:02.954934 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:02.954903 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7cvm2" event={"ID":"05683773-8c40-4d7c-9cc9-ee9d71b25b23","Type":"ContainerStarted","Data":"7128aa9d5a1adaf617fba25401a7ad241df89d83ac54c5988330bd696430379e"} Nov 03 20:48:02.955069 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:02.954961 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7cvm2" event={"ID":"05683773-8c40-4d7c-9cc9-ee9d71b25b23","Type":"ContainerStarted","Data":"0b1060f9f6c7355899b02931ecc51bed1b2de2482026b7f9eda67c5ea7ad7f32"} Nov 03 20:48:02.955069 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:02.954993 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7cvm2" Nov 03 20:48:03.072272 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.072235 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6"] Nov 03 20:48:03.075489 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.075465 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6" Nov 03 20:48:03.124284 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.124227 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7vv52" podStartSLOduration=1.602462026 podStartE2EDuration="5.124213935s" totalImagesPullingTime="3.521751909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:03.074697678 +0000 UTC m=+49.936573106" watchObservedRunningTime="2026-11-03 20:48:03.124213935 +0000 UTC m=+49.986089363" Nov 03 20:48:03.130886 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.130780 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 20:48:03.130886 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.130780 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 20:48:03.130886 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.130870 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 20:48:03.131298 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.131274 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 20:48:03.131420 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.131319 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-lhhqb\"" Nov 03 20:48:03.175818 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.175783 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 20:48:03.176921 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.176897 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6"] Nov 03 20:48:03.187227 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.187191 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ffhc8"] Nov 03 20:48:03.189918 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.189898 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:03.194575 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.194550 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96"] Nov 03 20:48:03.197189 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.197162 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:03.206416 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.206368 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-7cvm2" podStartSLOduration=2.457942858 podStartE2EDuration="5.206350607s" totalImagesPullingTime="2.748407749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:03.205996026 +0000 UTC m=+50.067871455" watchObservedRunningTime="2026-11-03 20:48:03.206350607 +0000 UTC m=+50.068226037" Nov 03 20:48:03.244422 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.244381 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 20:48:03.244422 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.244420 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 20:48:03.244687 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.244497 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 20:48:03.244687 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.244497 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-r6h6b\"" Nov 03 20:48:03.250851 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.250823 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c77a4d5b-a0d3-468b-aa66-7d607794aba1-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-zbcg6\" (UID: \"c77a4d5b-a0d3-468b-aa66-7d607794aba1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6" Nov 03 20:48:03.250994 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.250871 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c77a4d5b-a0d3-468b-aa66-7d607794aba1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-zbcg6\" (UID: \"c77a4d5b-a0d3-468b-aa66-7d607794aba1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6" Nov 03 20:48:03.250994 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.250895 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c77a4d5b-a0d3-468b-aa66-7d607794aba1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-zbcg6\" (UID: \"c77a4d5b-a0d3-468b-aa66-7d607794aba1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6" Nov 03 20:48:03.251086 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.250983 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pgk2\" (UniqueName: \"kubernetes.io/projected/c77a4d5b-a0d3-468b-aa66-7d607794aba1-kube-api-access-9pgk2\") pod \"openshift-state-metrics-7bd7489957-zbcg6\" (UID: \"c77a4d5b-a0d3-468b-aa66-7d607794aba1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6" Nov 03 20:48:03.252924 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.252891 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 20:48:03.254720 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.254696 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96"] Nov 03 20:48:03.274611 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.274586 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4vnfl\"" Nov 03 20:48:03.275156 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.275131 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 20:48:03.345090 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.345049 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 20:48:03.352391 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.352361 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3a695777-f42a-4cb6-8a31-c07c8daf321a-node-exporter-accelerators-collector-config\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:03.352553 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.352395 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-z8p96\" (UID: \"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:03.352553 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.352417 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-z8p96\" (UID: \"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:03.352553 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.352496 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbvc2\" (UniqueName: \"kubernetes.io/projected/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-kube-api-access-fbvc2\") pod \"kube-state-metrics-5dc8d7d89b-z8p96\" (UID: \"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:03.352553 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.352537 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a695777-f42a-4cb6-8a31-c07c8daf321a-sys\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:03.352753 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.352574 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c77a4d5b-a0d3-468b-aa66-7d607794aba1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-zbcg6\" (UID: \"c77a4d5b-a0d3-468b-aa66-7d607794aba1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6" Nov 03 20:48:03.352753 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.352625 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c77a4d5b-a0d3-468b-aa66-7d607794aba1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-zbcg6\" (UID: \"c77a4d5b-a0d3-468b-aa66-7d607794aba1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6" Nov 03 20:48:03.352753 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.352687 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3a695777-f42a-4cb6-8a31-c07c8daf321a-node-exporter-tls\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:03.352753 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.352724 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3a695777-f42a-4cb6-8a31-c07c8daf321a-node-exporter-textfile\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:03.352941 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.352753 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a695777-f42a-4cb6-8a31-c07c8daf321a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:03.352941 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.352797 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c77a4d5b-a0d3-468b-aa66-7d607794aba1-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-zbcg6\" (UID: \"c77a4d5b-a0d3-468b-aa66-7d607794aba1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6" Nov 03 20:48:03.352941 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.352824 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-z8p96\" (UID: \"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:03.352941 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.352860 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pgk2\" (UniqueName: \"kubernetes.io/projected/c77a4d5b-a0d3-468b-aa66-7d607794aba1-kube-api-access-9pgk2\") pod \"openshift-state-metrics-7bd7489957-zbcg6\" (UID: \"c77a4d5b-a0d3-468b-aa66-7d607794aba1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6" Nov 03 20:48:03.352941 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.352882 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a695777-f42a-4cb6-8a31-c07c8daf321a-metrics-client-ca\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:03.352941 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.352921 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-z8p96\" (UID: \"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:03.353211 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.352945 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhkvl\" (UniqueName: \"kubernetes.io/projected/3a695777-f42a-4cb6-8a31-c07c8daf321a-kube-api-access-lhkvl\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:03.353211 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.352977 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-z8p96\" (UID: \"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:03.353211 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.353000 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3a695777-f42a-4cb6-8a31-c07c8daf321a-node-exporter-wtmp\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:03.353211 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.353032 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3a695777-f42a-4cb6-8a31-c07c8daf321a-root\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:03.353847 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.353822 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c77a4d5b-a0d3-468b-aa66-7d607794aba1-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-zbcg6\" (UID: \"c77a4d5b-a0d3-468b-aa66-7d607794aba1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6" Nov 03 20:48:03.356651 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.356623 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c77a4d5b-a0d3-468b-aa66-7d607794aba1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-zbcg6\" (UID: \"c77a4d5b-a0d3-468b-aa66-7d607794aba1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6" Nov 03 20:48:03.356780 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.356650 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c77a4d5b-a0d3-468b-aa66-7d607794aba1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-zbcg6\" (UID: \"c77a4d5b-a0d3-468b-aa66-7d607794aba1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6" Nov 03 20:48:03.420636 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.420604 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pgk2\" (UniqueName: \"kubernetes.io/projected/c77a4d5b-a0d3-468b-aa66-7d607794aba1-kube-api-access-9pgk2\") pod \"openshift-state-metrics-7bd7489957-zbcg6\" (UID: \"c77a4d5b-a0d3-468b-aa66-7d607794aba1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6" Nov 03 20:48:03.453964 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.453930 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3a695777-f42a-4cb6-8a31-c07c8daf321a-node-exporter-tls\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:03.454145 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.453974 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3a695777-f42a-4cb6-8a31-c07c8daf321a-node-exporter-textfile\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:03.454224 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.454145 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a695777-f42a-4cb6-8a31-c07c8daf321a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:03.454280 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.454214 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-z8p96\" (UID: \"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:03.454280 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.454261 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a695777-f42a-4cb6-8a31-c07c8daf321a-metrics-client-ca\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:03.454385 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.454300 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-z8p96\" (UID: \"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:03.454385 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.454303 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3a695777-f42a-4cb6-8a31-c07c8daf321a-node-exporter-textfile\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:03.454385 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.454326 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhkvl\" (UniqueName: \"kubernetes.io/projected/3a695777-f42a-4cb6-8a31-c07c8daf321a-kube-api-access-lhkvl\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:03.454385 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.454360 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-z8p96\" (UID: \"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:03.454569 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.454386 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3a695777-f42a-4cb6-8a31-c07c8daf321a-node-exporter-wtmp\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:03.454569 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.454420 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3a695777-f42a-4cb6-8a31-c07c8daf321a-root\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:03.454569 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.454480 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3a695777-f42a-4cb6-8a31-c07c8daf321a-node-exporter-accelerators-collector-config\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:03.454569 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.454509 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-z8p96\" (UID: \"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:03.454569 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.454539 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-z8p96\" (UID: \"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:03.454569 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.454567 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbvc2\" (UniqueName: \"kubernetes.io/projected/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-kube-api-access-fbvc2\") pod \"kube-state-metrics-5dc8d7d89b-z8p96\" (UID: \"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:03.454855 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.454589 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a695777-f42a-4cb6-8a31-c07c8daf321a-sys\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:03.454855 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.454700 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a695777-f42a-4cb6-8a31-c07c8daf321a-sys\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:03.454855 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.454837 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3a695777-f42a-4cb6-8a31-c07c8daf321a-node-exporter-wtmp\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:03.454994 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.454897 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3a695777-f42a-4cb6-8a31-c07c8daf321a-root\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:03.455160 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.455139 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-z8p96\" (UID: \"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:03.455247 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.455222 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-z8p96\" (UID: \"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:03.455412 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.455393 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3a695777-f42a-4cb6-8a31-c07c8daf321a-node-exporter-accelerators-collector-config\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:03.455469 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.455442 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a695777-f42a-4cb6-8a31-c07c8daf321a-metrics-client-ca\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:03.455642 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.455625 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-z8p96\" (UID: \"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:03.457434 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.457403 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a695777-f42a-4cb6-8a31-c07c8daf321a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:03.457552 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.457441 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-z8p96\" (UID: \"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:03.457796 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.457775 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3a695777-f42a-4cb6-8a31-c07c8daf321a-node-exporter-tls\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:03.458766 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.458743 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-z8p96\" (UID: \"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:03.517647 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.517609 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhkvl\" (UniqueName: \"kubernetes.io/projected/3a695777-f42a-4cb6-8a31-c07c8daf321a-kube-api-access-lhkvl\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:03.517799 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.517690 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbvc2\" (UniqueName: \"kubernetes.io/projected/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-kube-api-access-fbvc2\") pod \"kube-state-metrics-5dc8d7d89b-z8p96\" (UID: \"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:03.693574 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.693539 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6" Nov 03 20:48:03.805917 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.805896 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:03.812901 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.812595 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:03.815699 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:48:03.815674 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a695777_f42a_4cb6_8a31_c07c8daf321a.slice/crio-bda995bdc42eaca76d1359b111bfb323e63e37da83761d452d97498210ee75c5 WatchSource:0}: Error finding container bda995bdc42eaca76d1359b111bfb323e63e37da83761d452d97498210ee75c5: Status 404 returned error can't find the container with id bda995bdc42eaca76d1359b111bfb323e63e37da83761d452d97498210ee75c5 Nov 03 20:48:03.958183 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.958102 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ffhc8" event={"ID":"3a695777-f42a-4cb6-8a31-c07c8daf321a","Type":"ContainerStarted","Data":"bda995bdc42eaca76d1359b111bfb323e63e37da83761d452d97498210ee75c5"} Nov 03 20:48:03.960134 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:03.960109 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tms4q" event={"ID":"e64dc44d-5470-4abb-8155-fdf7f24c253b","Type":"ContainerStarted","Data":"38c1e22499ecded9277bf71840dd2786bb7bd7856e1f4ec0b26c1bfa2e474c9c"} Nov 03 20:48:04.082065 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:04.082011 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tms4q" podStartSLOduration=1.7433511130000001 podStartE2EDuration="6.081995608s" totalImagesPullingTime="4.338644495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:47:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:04.081214121 +0000 UTC m=+50.943089549" watchObservedRunningTime="2026-11-03 20:48:04.081995608 +0000 UTC m=+50.943871035" Nov 03 20:48:04.111522 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:48:04.111494 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc77a4d5b_a0d3_468b_aa66_7d607794aba1.slice/crio-92de2ebb594e68d825289f2412232e1a91a62be2e243281001e49b04488d85ad WatchSource:0}: Error finding container 92de2ebb594e68d825289f2412232e1a91a62be2e243281001e49b04488d85ad: Status 404 returned error can't find the container with id 92de2ebb594e68d825289f2412232e1a91a62be2e243281001e49b04488d85ad Nov 03 20:48:04.114196 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:04.114175 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6"] Nov 03 20:48:04.188938 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:48:04.188911 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc14bf8f0_1335_4ee0_b49d_5d1f2836de1c.slice/crio-a1513668b128288d80b165d2599ebd99184addb467d8ed53c0f247d9e02a4af6 WatchSource:0}: Error finding container a1513668b128288d80b165d2599ebd99184addb467d8ed53c0f247d9e02a4af6: Status 404 returned error can't find the container with id a1513668b128288d80b165d2599ebd99184addb467d8ed53c0f247d9e02a4af6 Nov 03 20:48:04.189219 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:04.189198 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96"] Nov 03 20:48:04.967744 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:04.967694 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" event={"ID":"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c","Type":"ContainerStarted","Data":"a1513668b128288d80b165d2599ebd99184addb467d8ed53c0f247d9e02a4af6"} Nov 03 20:48:04.969607 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:04.969577 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6" event={"ID":"c77a4d5b-a0d3-468b-aa66-7d607794aba1","Type":"ContainerStarted","Data":"5467c14c682379d980a35e921f11c51d2260ddd00f95127ef982850b59deaacb"} Nov 03 20:48:04.969719 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:04.969629 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6" event={"ID":"c77a4d5b-a0d3-468b-aa66-7d607794aba1","Type":"ContainerStarted","Data":"5c319d2a4287125c03a33b0f578dfa785a6d5c676db9922a46c08833f61528fe"} Nov 03 20:48:04.969719 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:04.969647 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6" event={"ID":"c77a4d5b-a0d3-468b-aa66-7d607794aba1","Type":"ContainerStarted","Data":"92de2ebb594e68d825289f2412232e1a91a62be2e243281001e49b04488d85ad"} Nov 03 20:48:05.852123 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:05.852066 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5d578896-ftx7j"] Nov 03 20:48:05.854793 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:05.854774 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:05.909605 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:05.909573 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 20:48:05.909791 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:05.909713 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-86be21572r5l4\"" Nov 03 20:48:05.909854 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:05.909806 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 20:48:05.909912 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:05.909854 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 20:48:05.909978 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:05.909960 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 20:48:05.910157 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:05.910056 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-bst6w\"" Nov 03 20:48:05.910157 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:05.910116 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 20:48:05.921120 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:05.921096 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5d578896-ftx7j"] Nov 03 20:48:05.973774 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:05.973738 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:05.974244 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:05.973825 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:05.974244 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:05.973863 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-grpc-tls\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:05.974244 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:05.973964 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:05.974244 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:05.973999 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:05.974244 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:05.974034 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-thanos-querier-tls\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:05.974244 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:05.974092 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6x9d\" (UniqueName: \"kubernetes.io/projected/15a5eea7-15b5-45a6-838b-35914cb59be1-kube-api-access-x6x9d\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:05.974244 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:05.974204 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15a5eea7-15b5-45a6-838b-35914cb59be1-metrics-client-ca\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:05.974536 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:05.974261 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15a5eea7-15b5-45a6-838b-35914cb59be1-tmp\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:05.975444 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:05.975415 2575 generic.go:356] "Generic (PLEG): container finished" podID="3a695777-f42a-4cb6-8a31-c07c8daf321a" containerID="9d84ce16ea8218efd6a8292e83042ca841179f25f26f79f59eb17634644226b2" exitCode=0 Nov 03 20:48:05.975548 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:05.975513 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ffhc8" event={"ID":"3a695777-f42a-4cb6-8a31-c07c8daf321a","Type":"ContainerDied","Data":"9d84ce16ea8218efd6a8292e83042ca841179f25f26f79f59eb17634644226b2"} Nov 03 20:48:05.977964 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:05.977943 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6" event={"ID":"c77a4d5b-a0d3-468b-aa66-7d607794aba1","Type":"ContainerStarted","Data":"ddbd66891e0983f298c7de76803e331fa23ed88f5b843f0531d9f341e3183865"} Nov 03 20:48:06.074598 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.074572 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:06.074727 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.074624 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:06.074727 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.074650 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-grpc-tls\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:06.074842 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.074735 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:06.074842 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.074773 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:06.077698 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.075285 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-thanos-querier-tls\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:06.077698 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.075362 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6x9d\" (UniqueName: \"kubernetes.io/projected/15a5eea7-15b5-45a6-838b-35914cb59be1-kube-api-access-x6x9d\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:06.077698 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.075402 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15a5eea7-15b5-45a6-838b-35914cb59be1-metrics-client-ca\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:06.077698 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.075497 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15a5eea7-15b5-45a6-838b-35914cb59be1-tmp\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:06.077698 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.075875 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15a5eea7-15b5-45a6-838b-35914cb59be1-tmp\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:06.077698 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.077124 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15a5eea7-15b5-45a6-838b-35914cb59be1-metrics-client-ca\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:06.078344 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.078324 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:06.079014 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.078964 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:06.079108 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.079017 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:06.079266 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.079245 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-grpc-tls\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:06.079772 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.079752 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:06.080275 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.080251 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-thanos-querier-tls\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:06.140390 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.140365 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6x9d\" (UniqueName: \"kubernetes.io/projected/15a5eea7-15b5-45a6-838b-35914cb59be1-kube-api-access-x6x9d\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:06.173238 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.173210 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:06.242370 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.239030 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6" podStartSLOduration=1.646623852 podStartE2EDuration="3.23901025s" totalImagesPullingTime="1.592386398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:06.237135523 +0000 UTC m=+53.099010955" watchObservedRunningTime="2026-11-03 20:48:06.23901025 +0000 UTC m=+53.100885680" Nov 03 20:48:06.374044 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.373957 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f847c9697-nsrdm"] Nov 03 20:48:06.376320 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.376303 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:06.432741 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.432709 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 20:48:06.432741 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.432726 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 20:48:06.432741 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.432736 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 20:48:06.432960 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.432940 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 20:48:06.433000 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.432982 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 20:48:06.433118 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.433025 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 20:48:06.433118 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.433100 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-znh84\"" Nov 03 20:48:06.433417 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.433402 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 20:48:06.434259 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.434239 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f847c9697-nsrdm"] Nov 03 20:48:06.480005 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.479968 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-trusted-ca-bundle\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:06.480189 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.480018 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/195d22be-670c-4866-a303-1333085f8a67-tmp\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:06.480189 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.480130 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-oauth-serving-cert\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:06.480189 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.480162 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-service-ca\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:06.480305 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.480194 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-serving-cert\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:06.480305 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.480256 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-console-config\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:06.480305 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.480280 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw4cq\" (UniqueName: \"kubernetes.io/projected/195d22be-670c-4866-a303-1333085f8a67-kube-api-access-vw4cq\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:06.480397 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.480343 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-oauth-config\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:06.486200 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.486175 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 20:48:06.581611 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.581573 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw4cq\" (UniqueName: \"kubernetes.io/projected/195d22be-670c-4866-a303-1333085f8a67-kube-api-access-vw4cq\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:06.581611 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.581613 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-oauth-config\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:06.581865 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.581648 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-trusted-ca-bundle\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:06.581865 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.581683 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/195d22be-670c-4866-a303-1333085f8a67-tmp\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:06.581865 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.581715 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-oauth-serving-cert\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:06.581865 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.581749 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-service-ca\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:06.582060 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.581864 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-serving-cert\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:06.582060 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.581923 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-console-config\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:06.582338 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.582309 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/195d22be-670c-4866-a303-1333085f8a67-tmp\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:06.582723 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.582702 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-service-ca\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:06.582785 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.582764 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-console-config\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:06.582828 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.582779 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-oauth-serving-cert\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:06.583017 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.583001 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-trusted-ca-bundle\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:06.584280 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.584253 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-oauth-config\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:06.584362 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.584348 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-serving-cert\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:06.590449 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:48:06.590425 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a5eea7_15b5_45a6_838b_35914cb59be1.slice/crio-f6a2381d6d2ec12b671bc70f4b440c2c384ebc426ede6d9f412f720f3d3f1241 WatchSource:0}: Error finding container f6a2381d6d2ec12b671bc70f4b440c2c384ebc426ede6d9f412f720f3d3f1241: Status 404 returned error can't find the container with id f6a2381d6d2ec12b671bc70f4b440c2c384ebc426ede6d9f412f720f3d3f1241 Nov 03 20:48:06.591858 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.591839 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5d578896-ftx7j"] Nov 03 20:48:06.644471 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.644402 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw4cq\" (UniqueName: \"kubernetes.io/projected/195d22be-670c-4866-a303-1333085f8a67-kube-api-access-vw4cq\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:06.686469 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.686433 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:06.982832 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.982800 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" event={"ID":"15a5eea7-15b5-45a6-838b-35914cb59be1","Type":"ContainerStarted","Data":"f6a2381d6d2ec12b671bc70f4b440c2c384ebc426ede6d9f412f720f3d3f1241"} Nov 03 20:48:06.984985 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.984959 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" event={"ID":"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c","Type":"ContainerStarted","Data":"8f81ff8630e5757d86d169a73c6d99e8398f5468e4f85d73b8bb888d72eac2d6"} Nov 03 20:48:06.985132 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.985011 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" event={"ID":"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c","Type":"ContainerStarted","Data":"0022c87d80f4e9789f1f0081774babaf967d972172e2dfa9329dc50bde8e3a22"} Nov 03 20:48:06.985132 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.985036 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" event={"ID":"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c","Type":"ContainerStarted","Data":"a97df2863260eab18966174021e1585bf8a6b77998eecef18679a4adb77b1655"} Nov 03 20:48:06.987249 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.987221 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ffhc8" event={"ID":"3a695777-f42a-4cb6-8a31-c07c8daf321a","Type":"ContainerStarted","Data":"65d32a426b43d7c9db919addfb1fdb60e9b84a19758cce5b7bbc66d39fc659b5"} Nov 03 20:48:06.987352 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:06.987266 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ffhc8" event={"ID":"3a695777-f42a-4cb6-8a31-c07c8daf321a","Type":"ContainerStarted","Data":"ddfe0aa642190882e80af1b93d34d13dc5deaf1aa9cbaad55a393c4e15012072"} Nov 03 20:48:07.060835 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:48:07.060807 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195d22be_670c_4866_a303_1333085f8a67.slice/crio-e666b74d740ff3bc6f845fa19ccf562c1e0ee6a6f6475572c685ac7d72b6fbb3 WatchSource:0}: Error finding container e666b74d740ff3bc6f845fa19ccf562c1e0ee6a6f6475572c685ac7d72b6fbb3: Status 404 returned error can't find the container with id e666b74d740ff3bc6f845fa19ccf562c1e0ee6a6f6475572c685ac7d72b6fbb3 Nov 03 20:48:07.064259 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:07.064215 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f847c9697-nsrdm"] Nov 03 20:48:07.108558 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:07.108504 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" podStartSLOduration=2.228579503 podStartE2EDuration="4.108490317s" totalImagesPullingTime="1.879910814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:07.107767286 +0000 UTC m=+53.969642726" watchObservedRunningTime="2026-11-03 20:48:07.108490317 +0000 UTC m=+53.970365741" Nov 03 20:48:07.230766 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:07.230716 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ffhc8" podStartSLOduration=2.43878461 podStartE2EDuration="4.230701937s" totalImagesPullingTime="1.706893922s" totalInitContainerRuntime="85.023405ms" isStatefulPod=true podCreationTimestamp="2026-11-03 20:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:07.230498534 +0000 UTC m=+54.092373961" watchObservedRunningTime="2026-11-03 20:48:07.230701937 +0000 UTC m=+54.092577365" Nov 03 20:48:07.923745 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:07.923714 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-n8k2s"] Nov 03 20:48:07.926168 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:07.926141 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-n8k2s" Nov 03 20:48:07.980732 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:07.980700 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-k7jkq\"" Nov 03 20:48:07.980882 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:07.980787 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 20:48:07.981531 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:07.981508 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-n8k2s"] Nov 03 20:48:07.992231 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:07.992200 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f847c9697-nsrdm" event={"ID":"195d22be-670c-4866-a303-1333085f8a67","Type":"ContainerStarted","Data":"e666b74d740ff3bc6f845fa19ccf562c1e0ee6a6f6475572c685ac7d72b6fbb3"} Nov 03 20:48:08.045213 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.045177 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-b9b549fcc-4cxct"] Nov 03 20:48:08.047769 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.047747 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:08.095941 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.095906 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbhtz\" (UniqueName: \"kubernetes.io/projected/bee40e70-1cd3-4b89-8808-3abf1dcfbee9-kube-api-access-rbhtz\") pod \"monitoring-plugin-77f4f76594-n8k2s\" (UID: \"bee40e70-1cd3-4b89-8808-3abf1dcfbee9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-n8k2s" Nov 03 20:48:08.096146 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.096002 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bee40e70-1cd3-4b89-8808-3abf1dcfbee9-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-n8k2s\" (UID: \"bee40e70-1cd3-4b89-8808-3abf1dcfbee9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-n8k2s" Nov 03 20:48:08.105650 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.103387 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 20:48:08.105650 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.104195 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8bglh\"" Nov 03 20:48:08.105650 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.104235 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 20:48:08.105650 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.104309 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 20:48:08.105650 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.104418 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 20:48:08.105650 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.104992 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-kth16345cnsc\"" Nov 03 20:48:08.105650 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.105565 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-b9b549fcc-4cxct"] Nov 03 20:48:08.197536 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.197449 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbsdk\" (UniqueName: \"kubernetes.io/projected/ebf82283-d275-4640-bc44-5a461e4f9f9d-kube-api-access-pbsdk\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:08.197536 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.197522 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbhtz\" (UniqueName: \"kubernetes.io/projected/bee40e70-1cd3-4b89-8808-3abf1dcfbee9-kube-api-access-rbhtz\") pod \"monitoring-plugin-77f4f76594-n8k2s\" (UID: \"bee40e70-1cd3-4b89-8808-3abf1dcfbee9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-n8k2s" Nov 03 20:48:08.197717 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.197575 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ebf82283-d275-4640-bc44-5a461e4f9f9d-metrics-server-audit-profiles\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:08.197717 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.197616 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ebf82283-d275-4640-bc44-5a461e4f9f9d-audit-log\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:08.197717 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.197699 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ebf82283-d275-4640-bc44-5a461e4f9f9d-client-ca-bundle\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:08.197853 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.197772 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bee40e70-1cd3-4b89-8808-3abf1dcfbee9-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-n8k2s\" (UID: \"bee40e70-1cd3-4b89-8808-3abf1dcfbee9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-n8k2s" Nov 03 20:48:08.197900 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.197850 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebf82283-d275-4640-bc44-5a461e4f9f9d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:08.197945 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.197929 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ebf82283-d275-4640-bc44-5a461e4f9f9d-secret-metrics-server-tls\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:08.198020 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.198006 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ebf82283-d275-4640-bc44-5a461e4f9f9d-secret-metrics-server-client-certs\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:08.200490 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.200461 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bee40e70-1cd3-4b89-8808-3abf1dcfbee9-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-n8k2s\" (UID: \"bee40e70-1cd3-4b89-8808-3abf1dcfbee9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-n8k2s" Nov 03 20:48:08.260965 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.260927 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbhtz\" (UniqueName: \"kubernetes.io/projected/bee40e70-1cd3-4b89-8808-3abf1dcfbee9-kube-api-access-rbhtz\") pod \"monitoring-plugin-77f4f76594-n8k2s\" (UID: \"bee40e70-1cd3-4b89-8808-3abf1dcfbee9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-n8k2s" Nov 03 20:48:08.298878 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.298840 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ebf82283-d275-4640-bc44-5a461e4f9f9d-metrics-server-audit-profiles\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:08.299044 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.298888 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ebf82283-d275-4640-bc44-5a461e4f9f9d-audit-log\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:08.299316 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.299296 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ebf82283-d275-4640-bc44-5a461e4f9f9d-audit-log\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:08.299393 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.299348 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ebf82283-d275-4640-bc44-5a461e4f9f9d-client-ca-bundle\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:08.299449 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.299389 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebf82283-d275-4640-bc44-5a461e4f9f9d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:08.299497 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.299465 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ebf82283-d275-4640-bc44-5a461e4f9f9d-secret-metrics-server-tls\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:08.299547 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.299506 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ebf82283-d275-4640-bc44-5a461e4f9f9d-secret-metrics-server-client-certs\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:08.299599 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.299559 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbsdk\" (UniqueName: \"kubernetes.io/projected/ebf82283-d275-4640-bc44-5a461e4f9f9d-kube-api-access-pbsdk\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:08.299786 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.299761 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ebf82283-d275-4640-bc44-5a461e4f9f9d-metrics-server-audit-profiles\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:08.300222 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.300203 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebf82283-d275-4640-bc44-5a461e4f9f9d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:08.302723 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.302695 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ebf82283-d275-4640-bc44-5a461e4f9f9d-secret-metrics-server-tls\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:08.303015 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.302965 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ebf82283-d275-4640-bc44-5a461e4f9f9d-secret-metrics-server-client-certs\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:08.303154 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.303130 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ebf82283-d275-4640-bc44-5a461e4f9f9d-client-ca-bundle\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:08.363126 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.363092 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbsdk\" (UniqueName: \"kubernetes.io/projected/ebf82283-d275-4640-bc44-5a461e4f9f9d-kube-api-access-pbsdk\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:08.539409 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.539372 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-n8k2s" Nov 03 20:48:08.660934 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.660897 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:08.905688 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.905616 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f"] Nov 03 20:48:08.908276 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.908254 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:08.921548 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.921525 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-n8k2s"] Nov 03 20:48:08.954756 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:48:08.954718 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee40e70_1cd3_4b89_8808_3abf1dcfbee9.slice/crio-46ee947a9f7dbd243e535b5be14ff904040c24708f75e786f36c9ea6fe1b31e2 WatchSource:0}: Error finding container 46ee947a9f7dbd243e535b5be14ff904040c24708f75e786f36c9ea6fe1b31e2: Status 404 returned error can't find the container with id 46ee947a9f7dbd243e535b5be14ff904040c24708f75e786f36c9ea6fe1b31e2 Nov 03 20:48:08.963485 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.963454 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 03 20:48:08.963594 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.963466 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 03 20:48:08.963651 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.963599 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 03 20:48:08.963985 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.963730 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 03 20:48:08.963985 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.963899 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 03 20:48:08.963985 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.963952 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-cbbfr\"" Nov 03 20:48:08.997617 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:08.997547 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-n8k2s" event={"ID":"bee40e70-1cd3-4b89-8808-3abf1dcfbee9","Type":"ContainerStarted","Data":"46ee947a9f7dbd243e535b5be14ff904040c24708f75e786f36c9ea6fe1b31e2"} Nov 03 20:48:09.016217 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:09.016057 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 03 20:48:09.017661 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:09.017630 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f"] Nov 03 20:48:09.064707 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:09.064679 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-b9b549fcc-4cxct"] Nov 03 20:48:09.064846 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:48:09.064812 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf82283_d275_4640_bc44_5a461e4f9f9d.slice/crio-9f1dde26a34e0e5d20517e130b74b6c274b1d6a431637f4a41083294a50b2e2a WatchSource:0}: Error finding container 9f1dde26a34e0e5d20517e130b74b6c274b1d6a431637f4a41083294a50b2e2a: Status 404 returned error can't find the container with id 9f1dde26a34e0e5d20517e130b74b6c274b1d6a431637f4a41083294a50b2e2a Nov 03 20:48:09.106640 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:09.106523 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0327c758-d1f5-481d-9cf2-96e4b5f27872-telemeter-client-tls\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:09.106640 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:09.106592 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0327c758-d1f5-481d-9cf2-96e4b5f27872-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:09.106858 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:09.106676 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7rds\" (UniqueName: \"kubernetes.io/projected/0327c758-d1f5-481d-9cf2-96e4b5f27872-kube-api-access-p7rds\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:09.106858 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:09.106726 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0327c758-d1f5-481d-9cf2-96e4b5f27872-secret-telemeter-client\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:09.106858 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:09.106808 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0327c758-d1f5-481d-9cf2-96e4b5f27872-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:09.106858 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:09.106850 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0327c758-d1f5-481d-9cf2-96e4b5f27872-federate-client-tls\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:09.107046 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:09.106892 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0327c758-d1f5-481d-9cf2-96e4b5f27872-serving-certs-ca-bundle\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:09.107046 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:09.107039 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0327c758-d1f5-481d-9cf2-96e4b5f27872-metrics-client-ca\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:09.208997 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:09.208188 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7rds\" (UniqueName: \"kubernetes.io/projected/0327c758-d1f5-481d-9cf2-96e4b5f27872-kube-api-access-p7rds\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:09.208997 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:09.208246 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0327c758-d1f5-481d-9cf2-96e4b5f27872-secret-telemeter-client\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:09.208997 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:09.208287 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0327c758-d1f5-481d-9cf2-96e4b5f27872-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:09.208997 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:09.208319 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0327c758-d1f5-481d-9cf2-96e4b5f27872-federate-client-tls\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:09.208997 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:09.208361 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0327c758-d1f5-481d-9cf2-96e4b5f27872-serving-certs-ca-bundle\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:09.208997 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:09.208430 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0327c758-d1f5-481d-9cf2-96e4b5f27872-metrics-client-ca\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:09.208997 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:09.208461 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0327c758-d1f5-481d-9cf2-96e4b5f27872-telemeter-client-tls\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:09.208997 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:09.208490 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0327c758-d1f5-481d-9cf2-96e4b5f27872-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:09.211523 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:09.210725 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0327c758-d1f5-481d-9cf2-96e4b5f27872-serving-certs-ca-bundle\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:09.211523 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:09.211470 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0327c758-d1f5-481d-9cf2-96e4b5f27872-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:09.212926 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:09.212504 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0327c758-d1f5-481d-9cf2-96e4b5f27872-metrics-client-ca\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:09.212926 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:09.212893 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0327c758-d1f5-481d-9cf2-96e4b5f27872-federate-client-tls\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:09.213689 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:09.213638 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0327c758-d1f5-481d-9cf2-96e4b5f27872-secret-telemeter-client\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:09.213689 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:09.213649 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0327c758-d1f5-481d-9cf2-96e4b5f27872-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:09.214467 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:09.214444 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0327c758-d1f5-481d-9cf2-96e4b5f27872-telemeter-client-tls\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:09.280118 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:09.280089 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7rds\" (UniqueName: \"kubernetes.io/projected/0327c758-d1f5-481d-9cf2-96e4b5f27872-kube-api-access-p7rds\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:09.525410 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:09.525374 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:09.998619 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:09.998584 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f"] Nov 03 20:48:10.002829 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:10.002797 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" event={"ID":"ebf82283-d275-4640-bc44-5a461e4f9f9d","Type":"ContainerStarted","Data":"9f1dde26a34e0e5d20517e130b74b6c274b1d6a431637f4a41083294a50b2e2a"} Nov 03 20:48:10.005893 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:10.005862 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" event={"ID":"15a5eea7-15b5-45a6-838b-35914cb59be1","Type":"ContainerStarted","Data":"c4404bfd62987670fa1dd9cbfb636a5399c9b56babbccd786aa7b70b25113a2e"} Nov 03 20:48:10.006040 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:10.005943 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" event={"ID":"15a5eea7-15b5-45a6-838b-35914cb59be1","Type":"ContainerStarted","Data":"24ef1ec0ef90a7211b12467cdd3fe7903a1fc982704e67fcc290878ec9263c1b"} Nov 03 20:48:10.006040 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:10.005980 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" event={"ID":"15a5eea7-15b5-45a6-838b-35914cb59be1","Type":"ContainerStarted","Data":"4e0640e92a3bce90d8d908ace8aae32a44b5c92ab9b98f5af76748577c804106"} Nov 03 20:48:10.027298 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:48:10.027268 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0327c758_d1f5_481d_9cf2_96e4b5f27872.slice/crio-d2fdc7bebf927bd4cb7bd5a1e9fc6710f63867dfc7a957648e812082424e6b28 WatchSource:0}: Error finding container d2fdc7bebf927bd4cb7bd5a1e9fc6710f63867dfc7a957648e812082424e6b28: Status 404 returned error can't find the container with id d2fdc7bebf927bd4cb7bd5a1e9fc6710f63867dfc7a957648e812082424e6b28 Nov 03 20:48:11.014417 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:11.014378 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" event={"ID":"0327c758-d1f5-481d-9cf2-96e4b5f27872","Type":"ContainerStarted","Data":"d2fdc7bebf927bd4cb7bd5a1e9fc6710f63867dfc7a957648e812082424e6b28"} Nov 03 20:48:11.016513 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:11.016485 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f847c9697-nsrdm" event={"ID":"195d22be-670c-4866-a303-1333085f8a67","Type":"ContainerStarted","Data":"bac540ca14ec158883fadfe47d0dc4e0e5bfbd2b22dc73bb9cfe9cf7e95659d5"} Nov 03 20:48:11.073021 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:11.072992 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:11.140401 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:11.140302 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7f847c9697-nsrdm" podStartSLOduration=1.316939559 podStartE2EDuration="5.140284201s" totalImagesPullingTime="3.823344642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:11.139377112 +0000 UTC m=+58.001252539" watchObservedRunningTime="2026-11-03 20:48:11.140284201 +0000 UTC m=+58.002159630" Nov 03 20:48:12.021769 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:12.021722 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" event={"ID":"ebf82283-d275-4640-bc44-5a461e4f9f9d","Type":"ContainerStarted","Data":"2f5c075148fec907b6780d96f3d5f065751c3b3b58f86ec3b3dc030552a2b778"} Nov 03 20:48:12.025459 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:12.025424 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" event={"ID":"15a5eea7-15b5-45a6-838b-35914cb59be1","Type":"ContainerStarted","Data":"4dc67e5641ae4d0972215dac0b8f920a4b1b3876fe8bf26c033e72828cf7ac75"} Nov 03 20:48:12.025606 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:12.025510 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" event={"ID":"15a5eea7-15b5-45a6-838b-35914cb59be1","Type":"ContainerStarted","Data":"b75f1bd2d5188e571b58cc8af60c29243b130e29cb600ea3340a555052ed44c0"} Nov 03 20:48:12.025606 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:12.025552 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" event={"ID":"15a5eea7-15b5-45a6-838b-35914cb59be1","Type":"ContainerStarted","Data":"4e2c7f52c99def8babef188373fa75ae5c5e53557eaa5678582ab9b62822a63d"} Nov 03 20:48:12.025606 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:12.025597 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:12.027189 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:12.027155 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-n8k2s" event={"ID":"bee40e70-1cd3-4b89-8808-3abf1dcfbee9","Type":"ContainerStarted","Data":"702dbd5a3922239cd1cf23a368629bb6d3f192b1cb9898c714600267644d9e72"} Nov 03 20:48:12.027375 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:12.027354 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-n8k2s" Nov 03 20:48:12.032992 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:12.032971 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-n8k2s" Nov 03 20:48:12.153255 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:12.153199 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" podStartSLOduration=2.899172922 podStartE2EDuration="5.15318488s" totalImagesPullingTime="2.254011958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:12.151863704 +0000 UTC m=+59.013739132" watchObservedRunningTime="2026-11-03 20:48:12.15318488 +0000 UTC m=+59.015060308" Nov 03 20:48:12.280767 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:12.280723 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" podStartSLOduration=2.963367401 podStartE2EDuration="7.280697738s" totalImagesPullingTime="4.317330337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:12.28006469 +0000 UTC m=+59.141940122" watchObservedRunningTime="2026-11-03 20:48:12.280697738 +0000 UTC m=+59.142573166" Nov 03 20:48:12.402963 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:12.402905 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-n8k2s" podStartSLOduration=3.322047859 podStartE2EDuration="5.402890338s" totalImagesPullingTime="2.080842479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:12.400879584 +0000 UTC m=+59.262755013" watchObservedRunningTime="2026-11-03 20:48:12.402890338 +0000 UTC m=+59.264765766" Nov 03 20:48:12.963180 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:12.963145 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7cvm2" Nov 03 20:48:13.031818 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:13.031776 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" event={"ID":"0327c758-d1f5-481d-9cf2-96e4b5f27872","Type":"ContainerStarted","Data":"5f19a564b8f6dd0df19d416aba2d5fc388d00c556bb1c0d42997fee37f267ed9"} Nov 03 20:48:14.037060 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:14.037020 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" event={"ID":"0327c758-d1f5-481d-9cf2-96e4b5f27872","Type":"ContainerStarted","Data":"fddff2f29a59a2d6fd84a7cb8df00af1ef8af9986580e6d3dae32851cc4a6f9d"} Nov 03 20:48:14.037574 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:14.037138 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" event={"ID":"0327c758-d1f5-481d-9cf2-96e4b5f27872","Type":"ContainerStarted","Data":"2d356f3227c4a59354372c740fed3f9f39559f32acc72ffd79c288fce0a39d98"} Nov 03 20:48:14.318607 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:14.318555 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" podStartSLOduration=2.557120274 podStartE2EDuration="6.318540963s" totalImagesPullingTime="3.761420689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:14.266013852 +0000 UTC m=+61.127889280" watchObservedRunningTime="2026-11-03 20:48:14.318540963 +0000 UTC m=+61.180416391" Nov 03 20:48:15.425423 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:15.425388 2575 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f847c9697-nsrdm"] Nov 03 20:48:16.686816 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:16.686779 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:18.038306 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:18.038279 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:19.508777 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:19.508738 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-metrics-certs\") pod \"network-metrics-daemon-tqhnr\" (UID: \"c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c\") " pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:48:19.562480 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:19.562449 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 20:48:19.571740 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:19.571717 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-metrics-certs\") pod \"network-metrics-daemon-tqhnr\" (UID: \"c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c\") " pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:48:19.609832 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:19.609793 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5mgg\" (UniqueName: \"kubernetes.io/projected/955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81-kube-api-access-j5mgg\") pod \"network-check-target-lm85p\" (UID: \"955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81\") " pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:48:19.609832 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:19.609839 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/712fcecb-4757-4dd0-8f96-084b1e6ee903-original-pull-secret\") pod \"global-pull-secret-syncer-mqmds\" (UID: \"712fcecb-4757-4dd0-8f96-084b1e6ee903\") " pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:48:19.663429 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:19.663396 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 20:48:19.663599 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:19.663515 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 20:48:19.672811 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:19.672785 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/712fcecb-4757-4dd0-8f96-084b1e6ee903-original-pull-secret\") pod \"global-pull-secret-syncer-mqmds\" (UID: \"712fcecb-4757-4dd0-8f96-084b1e6ee903\") " pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:48:19.725456 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:19.725426 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 20:48:19.733471 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:19.733450 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5mgg\" (UniqueName: \"kubernetes.io/projected/955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81-kube-api-access-j5mgg\") pod \"network-check-target-lm85p\" (UID: \"955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81\") " pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:48:19.778763 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:19.778683 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l9gsz\"" Nov 03 20:48:19.782797 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:19.782773 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-snm99\"" Nov 03 20:48:19.786086 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:19.786054 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:48:19.789554 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:19.789539 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:48:20.062058 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:20.061987 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-78pt4\"" Nov 03 20:48:20.069131 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:20.069107 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:48:20.156857 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:48:20.156828 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod712fcecb_4757_4dd0_8f96_084b1e6ee903.slice/crio-6290d19f6ae8e83908c9bafdc5c1ea7f8b285d15954b85cbf70a2f1de6065d98 WatchSource:0}: Error finding container 6290d19f6ae8e83908c9bafdc5c1ea7f8b285d15954b85cbf70a2f1de6065d98: Status 404 returned error can't find the container with id 6290d19f6ae8e83908c9bafdc5c1ea7f8b285d15954b85cbf70a2f1de6065d98 Nov 03 20:48:20.159094 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:20.159054 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mqmds"] Nov 03 20:48:20.202968 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:48:20.202938 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ff8dbb_c78e_4bed_9f92_279c34c0ee9c.slice/crio-8c6bb37beee1291cde5d34c9079e670822933507e30b67380f8d3ff64a3f38ee WatchSource:0}: Error finding container 8c6bb37beee1291cde5d34c9079e670822933507e30b67380f8d3ff64a3f38ee: Status 404 returned error can't find the container with id 8c6bb37beee1291cde5d34c9079e670822933507e30b67380f8d3ff64a3f38ee Nov 03 20:48:20.205594 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:20.205575 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tqhnr"] Nov 03 20:48:20.434579 ip-10-0-129-192 kubenswrapper[2575]: W1103 20:48:20.434543 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod955dbaeb_db8a_4a1a_ba32_2ee7e8a7ad81.slice/crio-90cc044bb1c64adc17302b5dbc017fc1bb157884b0ff69d27f0574d5fc61d689 WatchSource:0}: Error finding container 90cc044bb1c64adc17302b5dbc017fc1bb157884b0ff69d27f0574d5fc61d689: Status 404 returned error can't find the container with id 90cc044bb1c64adc17302b5dbc017fc1bb157884b0ff69d27f0574d5fc61d689 Nov 03 20:48:20.436111 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:20.436045 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lm85p"] Nov 03 20:48:21.063330 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:21.063290 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mqmds" event={"ID":"712fcecb-4757-4dd0-8f96-084b1e6ee903","Type":"ContainerStarted","Data":"6290d19f6ae8e83908c9bafdc5c1ea7f8b285d15954b85cbf70a2f1de6065d98"} Nov 03 20:48:21.064582 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:21.064557 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tqhnr" event={"ID":"c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c","Type":"ContainerStarted","Data":"8c6bb37beee1291cde5d34c9079e670822933507e30b67380f8d3ff64a3f38ee"} Nov 03 20:48:21.065584 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:21.065566 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lm85p" event={"ID":"955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81","Type":"ContainerStarted","Data":"90cc044bb1c64adc17302b5dbc017fc1bb157884b0ff69d27f0574d5fc61d689"} Nov 03 20:48:22.072058 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:22.071963 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tqhnr" event={"ID":"c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c","Type":"ContainerStarted","Data":"d577989878e4429d2e5df16e64d43fc44b111c0616b1c1aeb83d55f537ab2cd5"} Nov 03 20:48:22.072058 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:22.072032 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tqhnr" event={"ID":"c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c","Type":"ContainerStarted","Data":"d9bcd4eb1748d0d01277d99c47d1d0c38b00d2cfe297b867fdc4be440d34af82"} Nov 03 20:48:22.194185 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:22.193902 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tqhnr" podStartSLOduration=66.597074049 podStartE2EDuration="1m8.193881048s" totalImagesPullingTime="1.596806999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:22.192087998 +0000 UTC m=+69.053963428" watchObservedRunningTime="2026-11-03 20:48:22.193881048 +0000 UTC m=+69.055756480" Nov 03 20:48:25.086239 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:25.086194 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lm85p" event={"ID":"955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81","Type":"ContainerStarted","Data":"b452da15acc74ced10866bfa3da80a6129feb766d886a07b283592327b7c5887"} Nov 03 20:48:25.086706 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:25.086314 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:48:25.206396 ip-10-0-129-192 kubenswrapper[2575]: I1103 20:48:25.206343 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lm85p" podStartSLOduration=67.568986663 podStartE2EDuration="1m11.206329329s" totalImagesPullingTime="3.637342666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:25.205610195 +0000 UTC m=+72.067485623" watchObservedRunningTime="2026-11-03 20:48:25.206329329 +0000 UTC m=+72.068204770" Nov 03 20:48:25.471678 ip-10-0-129-192 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 20:48:25.483670 ip-10-0-129-192 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 20:48:25.483863 ip-10-0-129-192 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 20:48:25.484033 ip-10-0-129-192 systemd[1]: kubelet.service: Consumed 4.272s CPU time. Nov 03 20:48:25.499720 ip-10-0-129-192 systemd[1]: Starting Kubernetes Kubelet... Nov 03 20:48:25.563760 ip-10-0-129-192 kubenswrapper[5615]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:48:25.563760 ip-10-0-129-192 kubenswrapper[5615]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 20:48:25.563760 ip-10-0-129-192 kubenswrapper[5615]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:48:25.563760 ip-10-0-129-192 kubenswrapper[5615]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:48:25.565947 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.565860 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:48:25.565947 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.565947 5615 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:48:25.566024 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.565952 5615 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:48:25.566024 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.565956 5615 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:48:25.566024 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.565959 5615 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:48:25.566024 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.565962 5615 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:48:25.566024 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.565964 5615 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:48:25.566024 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.565967 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:48:25.566024 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.565970 5615 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:48:25.566024 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.565973 5615 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:48:25.566024 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.565975 5615 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:48:25.566024 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.565978 5615 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:48:25.566024 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.565981 5615 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:48:25.566024 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.565983 5615 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:48:25.566024 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.565986 5615 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:48:25.566024 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.565989 5615 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:48:25.566024 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.565991 5615 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:48:25.566024 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.565994 5615 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:48:25.566024 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.565997 5615 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:48:25.566024 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.565999 5615 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:48:25.566024 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566002 5615 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:48:25.566024 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566005 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:48:25.566513 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566008 5615 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:48:25.566513 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566011 5615 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:48:25.566513 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566013 5615 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:48:25.566513 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566016 5615 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:48:25.566513 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566022 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:48:25.566513 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566025 5615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:48:25.566513 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566028 5615 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:48:25.566513 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566031 5615 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:48:25.566513 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566034 5615 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:48:25.566513 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566037 5615 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:48:25.566513 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566040 5615 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:48:25.566513 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566042 5615 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:48:25.566513 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566045 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:48:25.566513 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566047 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:48:25.566513 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566050 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:48:25.566513 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566053 5615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:48:25.566513 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566055 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:48:25.566513 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566058 5615 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:48:25.566513 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566061 5615 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:48:25.566975 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566064 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:48:25.566975 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566066 5615 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:48:25.566975 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566069 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:48:25.566975 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566086 5615 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:48:25.566975 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566089 5615 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:48:25.566975 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566091 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:48:25.566975 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566094 5615 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:48:25.566975 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566096 5615 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:48:25.566975 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566099 5615 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:48:25.566975 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566101 5615 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:48:25.566975 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566104 5615 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:48:25.566975 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566107 5615 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:48:25.566975 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566110 5615 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:48:25.566975 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566112 5615 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:48:25.566975 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566116 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:48:25.566975 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566119 5615 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:48:25.566975 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566122 5615 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:48:25.566975 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566124 5615 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:48:25.566975 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566128 5615 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:48:25.566975 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566130 5615 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:48:25.567465 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566133 5615 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:48:25.567465 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566136 5615 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:48:25.567465 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566139 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:48:25.567465 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566141 5615 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:48:25.567465 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566144 5615 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:48:25.567465 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566147 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:48:25.567465 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566150 5615 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:48:25.567465 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566153 5615 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:48:25.567465 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566155 5615 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:48:25.567465 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566158 5615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:48:25.567465 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566161 5615 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:48:25.567465 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566164 5615 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:48:25.567465 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566167 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:48:25.567465 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566170 5615 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:48:25.567465 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566174 5615 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:48:25.567465 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566177 5615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:48:25.567465 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566179 5615 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:48:25.567465 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566182 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:48:25.567928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566185 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:48:25.567928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566187 5615 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:48:25.567928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566190 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:48:25.567928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566192 5615 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:48:25.567928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566195 5615 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:48:25.567928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566197 5615 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:48:25.567928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566200 5615 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:48:25.567928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566203 5615 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:48:25.567928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566206 5615 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:48:25.567928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566209 5615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:48:25.567928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566212 5615 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:48:25.567928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566215 5615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:48:25.567928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566217 5615 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:48:25.567928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566221 5615 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:48:25.567928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566225 5615 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:48:25.567928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566229 5615 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:48:25.567928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566233 5615 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:48:25.567928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566236 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:48:25.567928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566238 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:48:25.568434 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566241 5615 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:48:25.568434 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566243 5615 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:48:25.568434 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566246 5615 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:48:25.568434 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566249 5615 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:48:25.568434 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566252 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:48:25.568434 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566255 5615 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:48:25.568434 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566259 5615 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:48:25.568434 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.566263 5615 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:48:25.568434 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567010 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:48:25.568434 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567018 5615 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:48:25.568434 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567020 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:48:25.568434 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567024 5615 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:48:25.568434 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567027 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:48:25.568434 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567030 5615 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:48:25.568434 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567033 5615 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:48:25.568434 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567035 5615 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:48:25.568434 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567038 5615 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:48:25.568434 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567043 5615 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:48:25.568434 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567047 5615 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:48:25.568890 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567050 5615 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:48:25.568890 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567054 5615 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:48:25.568890 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567057 5615 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:48:25.568890 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567060 5615 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:48:25.568890 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567064 5615 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:48:25.568890 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567067 5615 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:48:25.568890 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567086 5615 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:48:25.568890 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567091 5615 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:48:25.568890 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567096 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:48:25.568890 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567100 5615 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:48:25.568890 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567106 5615 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:48:25.568890 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567111 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:48:25.568890 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567115 5615 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:48:25.568890 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567118 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:48:25.568890 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567120 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:48:25.568890 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567128 5615 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:48:25.568890 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567131 5615 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:48:25.568890 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567134 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:48:25.568890 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567136 5615 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:48:25.568890 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567140 5615 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:48:25.569376 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567142 5615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:48:25.569376 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567145 5615 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:48:25.569376 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567148 5615 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:48:25.569376 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567151 5615 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:48:25.569376 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567155 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:48:25.569376 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567158 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:48:25.569376 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567160 5615 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:48:25.569376 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567163 5615 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:48:25.569376 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567166 5615 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:48:25.569376 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567168 5615 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:48:25.569376 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567171 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:48:25.569376 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567174 5615 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:48:25.569376 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567177 5615 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:48:25.569376 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567179 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:48:25.569376 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567183 5615 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:48:25.569376 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567186 5615 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:48:25.569376 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567188 5615 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:48:25.569376 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567191 5615 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:48:25.569376 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567194 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:48:25.569856 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567196 5615 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:48:25.569856 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567199 5615 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:48:25.569856 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567202 5615 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:48:25.569856 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567205 5615 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:48:25.569856 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567208 5615 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:48:25.569856 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567210 5615 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:48:25.569856 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567213 5615 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:48:25.569856 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567216 5615 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:48:25.569856 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567220 5615 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:48:25.569856 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567223 5615 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:48:25.569856 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567226 5615 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:48:25.569856 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567228 5615 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:48:25.569856 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567231 5615 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:48:25.569856 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567234 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:48:25.569856 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567236 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:48:25.569856 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567239 5615 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:48:25.569856 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567242 5615 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:48:25.569856 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567244 5615 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:48:25.569856 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567247 5615 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:48:25.569856 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567250 5615 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:48:25.570402 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567253 5615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:48:25.570402 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567255 5615 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:48:25.570402 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567258 5615 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:48:25.570402 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567261 5615 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:48:25.570402 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567264 5615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:48:25.570402 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567266 5615 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:48:25.570402 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567269 5615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:48:25.570402 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567273 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:48:25.570402 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567275 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:48:25.570402 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567278 5615 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:48:25.570402 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567280 5615 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:48:25.570402 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567283 5615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:48:25.570402 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567286 5615 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:48:25.570402 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567288 5615 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:48:25.570402 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567291 5615 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:48:25.570402 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567294 5615 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:48:25.570402 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567297 5615 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:48:25.570402 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567299 5615 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:48:25.570402 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567302 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:48:25.570402 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567305 5615 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:48:25.570912 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567309 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:48:25.570912 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567312 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:48:25.570912 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567315 5615 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:48:25.570912 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567318 5615 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:48:25.570912 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567321 5615 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:48:25.570912 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567324 5615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:48:25.570912 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567328 5615 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:48:25.570912 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567331 5615 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:48:25.570912 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567333 5615 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:48:25.570912 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567336 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:48:25.570912 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567338 5615 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:48:25.570912 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567341 5615 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:48:25.570912 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567344 5615 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:48:25.570912 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567346 5615 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:48:25.570912 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567349 5615 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:48:25.570912 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567351 5615 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:48:25.570912 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567852 5615 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:48:25.570912 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567857 5615 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:48:25.570912 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567860 5615 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:48:25.570912 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567863 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:48:25.571386 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567866 5615 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:48:25.571386 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567869 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:48:25.571386 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567871 5615 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:48:25.571386 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567874 5615 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:48:25.571386 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567877 5615 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:48:25.571386 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567880 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:48:25.571386 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567882 5615 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:48:25.571386 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567886 5615 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:48:25.571386 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567889 5615 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:48:25.571386 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567892 5615 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:48:25.571386 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567894 5615 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:48:25.571386 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567897 5615 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:48:25.571386 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567901 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:48:25.571386 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567903 5615 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:48:25.571386 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567906 5615 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:48:25.571386 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567908 5615 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:48:25.571386 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567912 5615 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:48:25.571386 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567915 5615 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:48:25.571386 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567918 5615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:48:25.571386 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567920 5615 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:48:25.571884 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567923 5615 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:48:25.571884 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567925 5615 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:48:25.571884 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567928 5615 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:48:25.571884 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567930 5615 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:48:25.571884 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567933 5615 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:48:25.571884 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567936 5615 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:48:25.571884 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567939 5615 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:48:25.571884 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567942 5615 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:48:25.571884 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567944 5615 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:48:25.571884 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567947 5615 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:48:25.571884 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567950 5615 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:48:25.571884 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567953 5615 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:48:25.571884 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567956 5615 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:48:25.571884 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567958 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:48:25.571884 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567961 5615 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:48:25.571884 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567964 5615 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:48:25.571884 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567966 5615 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:48:25.571884 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567969 5615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:48:25.571884 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567972 5615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:48:25.571884 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567974 5615 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:48:25.572367 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567978 5615 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:48:25.572367 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567980 5615 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:48:25.572367 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567984 5615 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:48:25.572367 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567987 5615 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:48:25.572367 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567990 5615 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:48:25.572367 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567993 5615 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:48:25.572367 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567996 5615 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:48:25.572367 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.567999 5615 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:48:25.572367 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568002 5615 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:48:25.572367 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568005 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:48:25.572367 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568008 5615 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:48:25.572367 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568011 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:48:25.572367 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568013 5615 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:48:25.572367 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568016 5615 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:48:25.572367 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568019 5615 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:48:25.572367 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568021 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:48:25.572367 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568024 5615 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:48:25.572367 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568027 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:48:25.572367 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568029 5615 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:48:25.572819 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568032 5615 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:48:25.572819 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568035 5615 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:48:25.572819 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568038 5615 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:48:25.572819 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568041 5615 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:48:25.572819 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568044 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:48:25.572819 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568046 5615 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:48:25.572819 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568049 5615 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:48:25.572819 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568052 5615 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:48:25.572819 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568054 5615 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:48:25.572819 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568057 5615 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:48:25.572819 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568060 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:48:25.572819 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568062 5615 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:48:25.572819 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568065 5615 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:48:25.572819 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568067 5615 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:48:25.572819 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568084 5615 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:48:25.572819 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568088 5615 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:48:25.572819 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568090 5615 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:48:25.572819 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568093 5615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:48:25.572819 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568096 5615 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:48:25.572819 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568098 5615 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:48:25.573416 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568101 5615 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:48:25.573416 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568106 5615 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:48:25.573416 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568109 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:48:25.573416 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568111 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:48:25.573416 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568114 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:48:25.573416 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568117 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:48:25.573416 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568119 5615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:48:25.573416 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568122 5615 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:48:25.573416 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568125 5615 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:48:25.573416 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568127 5615 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:48:25.573416 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568130 5615 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:48:25.573416 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568133 5615 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:48:25.573416 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568135 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:48:25.573416 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568138 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:48:25.573416 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568141 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:48:25.573416 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568146 5615 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:48:25.573416 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568150 5615 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:48:25.573416 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568153 5615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:48:25.573416 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568156 5615 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:48:25.574158 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568159 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:48:25.574158 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568162 5615 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:48:25.574158 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568164 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:48:25.574158 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568167 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:48:25.574158 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568264 5615 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 20:48:25.574158 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568271 5615 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 20:48:25.574158 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568278 5615 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 20:48:25.574158 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568282 5615 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 20:48:25.574158 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568288 5615 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 20:48:25.574158 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568291 5615 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 20:48:25.574158 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568295 5615 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 20:48:25.574158 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568299 5615 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 20:48:25.574158 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568302 5615 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 20:48:25.574158 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568306 5615 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 20:48:25.574158 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568312 5615 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 20:48:25.574158 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568315 5615 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 20:48:25.574158 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568320 5615 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 20:48:25.574158 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568323 5615 flags.go:64] FLAG: --cgroup-root="" Nov 03 20:48:25.574158 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568326 5615 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 20:48:25.574158 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568330 5615 flags.go:64] FLAG: --client-ca-file="" Nov 03 20:48:25.574158 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568333 5615 flags.go:64] FLAG: --cloud-provider="external" Nov 03 20:48:25.574158 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568336 5615 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 20:48:25.574158 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568340 5615 flags.go:64] FLAG: --cluster-domain="" Nov 03 20:48:25.574734 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568343 5615 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 20:48:25.574734 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568346 5615 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 20:48:25.574734 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568349 5615 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 20:48:25.574734 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568353 5615 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 20:48:25.574734 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568357 5615 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 20:48:25.574734 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568360 5615 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 20:48:25.574734 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568363 5615 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 20:48:25.574734 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568368 5615 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 20:48:25.574734 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568371 5615 flags.go:64] FLAG: --contention-profiling="false" Nov 03 20:48:25.574734 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568374 5615 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 20:48:25.574734 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568377 5615 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 20:48:25.574734 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568380 5615 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 20:48:25.574734 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568383 5615 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 20:48:25.574734 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568388 5615 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 20:48:25.574734 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568391 5615 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 20:48:25.574734 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568394 5615 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 20:48:25.574734 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568397 5615 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 20:48:25.574734 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568400 5615 flags.go:64] FLAG: --enable-server="true" Nov 03 20:48:25.574734 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568404 5615 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 20:48:25.574734 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568408 5615 flags.go:64] FLAG: --event-burst="100" Nov 03 20:48:25.574734 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568412 5615 flags.go:64] FLAG: --event-qps="50" Nov 03 20:48:25.574734 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568415 5615 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 20:48:25.574734 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568418 5615 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 20:48:25.574734 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568422 5615 flags.go:64] FLAG: --eviction-hard="" Nov 03 20:48:25.575321 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568426 5615 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 20:48:25.575321 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568431 5615 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 20:48:25.575321 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568435 5615 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 20:48:25.575321 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568438 5615 flags.go:64] FLAG: --eviction-soft="" Nov 03 20:48:25.575321 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568441 5615 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 20:48:25.575321 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568445 5615 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 20:48:25.575321 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568448 5615 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 20:48:25.575321 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568451 5615 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 20:48:25.575321 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568454 5615 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 20:48:25.575321 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568457 5615 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 20:48:25.575321 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568460 5615 flags.go:64] FLAG: --feature-gates="" Nov 03 20:48:25.575321 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568464 5615 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 20:48:25.575321 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568467 5615 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 20:48:25.575321 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568470 5615 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 20:48:25.575321 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568473 5615 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 20:48:25.575321 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568476 5615 flags.go:64] FLAG: --healthz-port="10248" Nov 03 20:48:25.575321 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568480 5615 flags.go:64] FLAG: --help="false" Nov 03 20:48:25.575321 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568483 5615 flags.go:64] FLAG: --hostname-override="ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:48:25.575321 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568486 5615 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 20:48:25.575321 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568489 5615 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 20:48:25.575321 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568492 5615 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 20:48:25.575321 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568495 5615 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 20:48:25.575321 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568499 5615 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 20:48:25.575868 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568502 5615 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 20:48:25.575868 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568505 5615 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 20:48:25.575868 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568508 5615 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 20:48:25.575868 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568511 5615 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 20:48:25.575868 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568514 5615 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 20:48:25.575868 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568518 5615 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 20:48:25.575868 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568521 5615 flags.go:64] FLAG: --kube-reserved="" Nov 03 20:48:25.575868 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568524 5615 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 20:48:25.575868 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568528 5615 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 20:48:25.575868 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568531 5615 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 20:48:25.575868 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568536 5615 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 20:48:25.575868 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568539 5615 flags.go:64] FLAG: --lock-file="" Nov 03 20:48:25.575868 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568542 5615 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 20:48:25.575868 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568545 5615 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 20:48:25.575868 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568548 5615 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 20:48:25.575868 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568554 5615 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 20:48:25.575868 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568557 5615 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 20:48:25.575868 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568560 5615 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 20:48:25.575868 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568563 5615 flags.go:64] FLAG: --logging-format="text" Nov 03 20:48:25.575868 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568566 5615 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 20:48:25.575868 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568569 5615 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 20:48:25.575868 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568573 5615 flags.go:64] FLAG: --manifest-url="" Nov 03 20:48:25.575868 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568576 5615 flags.go:64] FLAG: --manifest-url-header="" Nov 03 20:48:25.575868 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568580 5615 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 20:48:25.575868 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568584 5615 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 20:48:25.576501 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568587 5615 flags.go:64] FLAG: --max-pods="110" Nov 03 20:48:25.576501 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568591 5615 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 20:48:25.576501 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568594 5615 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 20:48:25.576501 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568597 5615 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 20:48:25.576501 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568600 5615 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 20:48:25.576501 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568603 5615 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 20:48:25.576501 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568606 5615 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 20:48:25.576501 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568609 5615 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 20:48:25.576501 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568616 5615 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 20:48:25.576501 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568619 5615 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 20:48:25.576501 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568622 5615 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 20:48:25.576501 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568625 5615 flags.go:64] FLAG: --pod-cidr="" Nov 03 20:48:25.576501 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568635 5615 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 20:48:25.576501 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568639 5615 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 20:48:25.576501 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568642 5615 flags.go:64] FLAG: --pods-per-core="0" Nov 03 20:48:25.576501 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568646 5615 flags.go:64] FLAG: --port="10250" Nov 03 20:48:25.576501 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568649 5615 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 20:48:25.576501 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568654 5615 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0eec255b93f603248" Nov 03 20:48:25.576501 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568657 5615 flags.go:64] FLAG: --qos-reserved="" Nov 03 20:48:25.576501 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568661 5615 flags.go:64] FLAG: --read-only-port="10255" Nov 03 20:48:25.576501 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568664 5615 flags.go:64] FLAG: --register-node="true" Nov 03 20:48:25.576501 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568667 5615 flags.go:64] FLAG: --register-with-taints="" Nov 03 20:48:25.576501 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568676 5615 flags.go:64] FLAG: --registry-burst="10" Nov 03 20:48:25.576501 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568679 5615 flags.go:64] FLAG: --registry-qps="5" Nov 03 20:48:25.576501 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568682 5615 flags.go:64] FLAG: --reserved-cpus="" Nov 03 20:48:25.576501 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568684 5615 flags.go:64] FLAG: --reserved-memory="" Nov 03 20:48:25.577126 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568689 5615 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 20:48:25.577126 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568692 5615 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 20:48:25.577126 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568695 5615 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 20:48:25.577126 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568698 5615 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 20:48:25.577126 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568702 5615 flags.go:64] FLAG: --runonce="false" Nov 03 20:48:25.577126 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568705 5615 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 20:48:25.577126 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568708 5615 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 20:48:25.577126 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568711 5615 flags.go:64] FLAG: --seccomp-default="false" Nov 03 20:48:25.577126 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568714 5615 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 20:48:25.577126 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568717 5615 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 20:48:25.577126 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568720 5615 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 20:48:25.577126 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568723 5615 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 20:48:25.577126 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568726 5615 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 20:48:25.577126 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568729 5615 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 20:48:25.577126 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568732 5615 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 20:48:25.577126 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568735 5615 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 20:48:25.577126 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568738 5615 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 20:48:25.577126 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568741 5615 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 20:48:25.577126 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568745 5615 flags.go:64] FLAG: --system-cgroups="" Nov 03 20:48:25.577126 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568748 5615 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 20:48:25.577126 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568754 5615 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 20:48:25.577126 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568759 5615 flags.go:64] FLAG: --tls-cert-file="" Nov 03 20:48:25.577126 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568762 5615 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 20:48:25.577126 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568767 5615 flags.go:64] FLAG: --tls-min-version="" Nov 03 20:48:25.577126 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568770 5615 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 20:48:25.577739 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568773 5615 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 20:48:25.577739 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568776 5615 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 20:48:25.577739 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568779 5615 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 20:48:25.577739 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568782 5615 flags.go:64] FLAG: --v="2" Nov 03 20:48:25.577739 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568787 5615 flags.go:64] FLAG: --version="false" Nov 03 20:48:25.577739 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568791 5615 flags.go:64] FLAG: --vmodule="" Nov 03 20:48:25.577739 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568796 5615 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 20:48:25.577739 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.568799 5615 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 20:48:25.577739 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568931 5615 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:48:25.577739 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568935 5615 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:48:25.577739 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568938 5615 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:48:25.577739 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568941 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:48:25.577739 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568944 5615 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:48:25.577739 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568947 5615 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:48:25.577739 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568950 5615 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:48:25.577739 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568953 5615 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:48:25.577739 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568955 5615 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:48:25.577739 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568958 5615 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:48:25.577739 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568961 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:48:25.577739 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568964 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:48:25.577739 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568966 5615 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:48:25.578254 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568969 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:48:25.578254 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568972 5615 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:48:25.578254 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568974 5615 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:48:25.578254 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568977 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:48:25.578254 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568980 5615 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:48:25.578254 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568982 5615 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:48:25.578254 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568985 5615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:48:25.578254 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568990 5615 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:48:25.578254 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568993 5615 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:48:25.578254 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568996 5615 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:48:25.578254 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.568999 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:48:25.578254 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569002 5615 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:48:25.578254 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569005 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:48:25.578254 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569008 5615 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:48:25.578254 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569010 5615 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:48:25.578254 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569013 5615 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:48:25.578254 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569016 5615 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:48:25.578254 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569019 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:48:25.578254 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569021 5615 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:48:25.578703 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569024 5615 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:48:25.578703 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569027 5615 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:48:25.578703 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569029 5615 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:48:25.578703 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569032 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:48:25.578703 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569034 5615 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:48:25.578703 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569037 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:48:25.578703 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569040 5615 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:48:25.578703 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569042 5615 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:48:25.578703 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569045 5615 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:48:25.578703 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569048 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:48:25.578703 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569051 5615 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:48:25.578703 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569053 5615 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:48:25.578703 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569056 5615 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:48:25.578703 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569059 5615 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:48:25.578703 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569062 5615 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:48:25.578703 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569064 5615 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:48:25.578703 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569067 5615 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:48:25.578703 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569069 5615 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:48:25.578703 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569086 5615 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:48:25.579167 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569090 5615 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:48:25.579167 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569096 5615 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:48:25.579167 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569099 5615 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:48:25.579167 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569103 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:48:25.579167 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569106 5615 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:48:25.579167 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569108 5615 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:48:25.579167 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569111 5615 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:48:25.579167 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569114 5615 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:48:25.579167 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569117 5615 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:48:25.579167 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569119 5615 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:48:25.579167 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569122 5615 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:48:25.579167 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569125 5615 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:48:25.579167 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569129 5615 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:48:25.579167 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569132 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:48:25.579167 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569135 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:48:25.579167 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569138 5615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:48:25.579167 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569141 5615 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:48:25.579167 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569144 5615 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:48:25.579167 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569146 5615 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:48:25.579167 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569149 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:48:25.579647 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569152 5615 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:48:25.579647 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569154 5615 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:48:25.579647 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569157 5615 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:48:25.579647 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569160 5615 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:48:25.579647 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569162 5615 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:48:25.579647 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569165 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:48:25.579647 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569167 5615 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:48:25.579647 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569170 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:48:25.579647 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569173 5615 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:48:25.579647 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569175 5615 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:48:25.579647 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569178 5615 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:48:25.579647 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569181 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:48:25.579647 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569183 5615 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:48:25.579647 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569187 5615 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:48:25.579647 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569190 5615 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:48:25.579647 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569194 5615 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:48:25.579647 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569197 5615 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:48:25.579647 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569200 5615 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:48:25.579647 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569202 5615 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:48:25.579647 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569205 5615 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:48:25.580235 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569208 5615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:48:25.580235 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569211 5615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:48:25.580235 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569213 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:48:25.580235 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569216 5615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:48:25.580235 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569219 5615 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:48:25.580235 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569221 5615 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:48:25.580235 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569224 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:48:25.580235 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569227 5615 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:48:25.580235 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569230 5615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:48:25.580235 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569232 5615 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:48:25.580235 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569235 5615 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:48:25.580235 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569238 5615 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:48:25.580235 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569240 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:48:25.580235 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569243 5615 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:48:25.580235 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569246 5615 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:48:25.580235 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.569271 5615 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:48:25.580671 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569307 5615 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:48:25.580671 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569311 5615 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:48:25.580671 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569314 5615 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:48:25.580671 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569317 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:48:25.580671 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569320 5615 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:48:25.580671 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569322 5615 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:48:25.580671 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569325 5615 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:48:25.580671 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569328 5615 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:48:25.580671 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569330 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:48:25.580671 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569335 5615 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:48:25.580671 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569338 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:48:25.580671 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569341 5615 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:48:25.580671 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569345 5615 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:48:25.580671 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569348 5615 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:48:25.580671 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569350 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:48:25.580671 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569353 5615 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:48:25.580671 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569356 5615 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:48:25.580671 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569358 5615 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:48:25.580671 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569361 5615 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:48:25.580671 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569364 5615 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:48:25.581380 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569367 5615 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:48:25.581380 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569369 5615 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:48:25.581380 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569372 5615 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:48:25.581380 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569376 5615 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:48:25.581380 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569380 5615 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:48:25.581380 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569384 5615 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:48:25.581380 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569388 5615 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:48:25.581380 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569392 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:48:25.581380 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569395 5615 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:48:25.581380 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569398 5615 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:48:25.581380 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569400 5615 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:48:25.581380 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569403 5615 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:48:25.581380 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569406 5615 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:48:25.581380 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569408 5615 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:48:25.581380 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569411 5615 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:48:25.581380 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569414 5615 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:48:25.581380 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569417 5615 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:48:25.581380 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569421 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:48:25.581380 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569424 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:48:25.581928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569427 5615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:48:25.581928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569430 5615 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:48:25.581928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569433 5615 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:48:25.581928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569436 5615 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:48:25.581928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569438 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:48:25.581928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569443 5615 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:48:25.581928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569446 5615 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:48:25.581928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569448 5615 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:48:25.581928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569451 5615 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:48:25.581928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569454 5615 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:48:25.581928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569457 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:48:25.581928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569459 5615 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:48:25.581928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569462 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:48:25.581928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569465 5615 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:48:25.581928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569468 5615 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:48:25.581928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569470 5615 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:48:25.581928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569473 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:48:25.581928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569475 5615 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:48:25.581928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569478 5615 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:48:25.581928 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569481 5615 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:48:25.582475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569484 5615 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:48:25.582475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569486 5615 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:48:25.582475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569489 5615 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:48:25.582475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569498 5615 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:48:25.582475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569503 5615 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:48:25.582475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569506 5615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:48:25.582475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569509 5615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:48:25.582475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569511 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:48:25.582475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569514 5615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:48:25.582475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569517 5615 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:48:25.582475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569519 5615 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:48:25.582475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569523 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:48:25.582475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569526 5615 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:48:25.582475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569529 5615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:48:25.582475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569531 5615 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:48:25.582475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569534 5615 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:48:25.582475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569537 5615 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:48:25.582475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569541 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:48:25.582475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569545 5615 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:48:25.582984 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569548 5615 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:48:25.582984 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569551 5615 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:48:25.582984 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569554 5615 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:48:25.582984 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569556 5615 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:48:25.582984 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569560 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:48:25.582984 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569562 5615 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:48:25.582984 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569565 5615 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:48:25.582984 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569568 5615 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:48:25.582984 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569570 5615 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:48:25.582984 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569573 5615 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:48:25.582984 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569576 5615 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:48:25.582984 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569578 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:48:25.582984 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569581 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:48:25.582984 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569584 5615 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:48:25.582984 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569586 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:48:25.582984 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569589 5615 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:48:25.582984 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569592 5615 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:48:25.582984 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569594 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:48:25.582984 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569597 5615 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:48:25.583475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569600 5615 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:48:25.583475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569603 5615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:48:25.583475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569605 5615 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:48:25.583475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569608 5615 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:48:25.583475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569610 5615 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:48:25.583475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569613 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:48:25.583475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569616 5615 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:48:25.583475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569618 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:48:25.583475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.569622 5615 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:48:25.583475 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.574571 5615 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 20:48:25.583475 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.574591 5615 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 20:48:25.583475 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.574613 5615 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 20:48:25.583475 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.574619 5615 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 20:48:25.583475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574679 5615 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:48:25.583475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574685 5615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:48:25.583475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574689 5615 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:48:25.583475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574692 5615 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:48:25.583475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574695 5615 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:48:25.583475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574698 5615 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:48:25.583475 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574701 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:48:25.583955 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574703 5615 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:48:25.583955 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574706 5615 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:48:25.583955 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574709 5615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:48:25.583955 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574712 5615 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:48:25.583955 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574715 5615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:48:25.583955 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574718 5615 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:48:25.583955 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574720 5615 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:48:25.583955 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574724 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:48:25.583955 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574730 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:48:25.583955 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574733 5615 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:48:25.583955 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574736 5615 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:48:25.583955 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574739 5615 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:48:25.583955 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574742 5615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:48:25.583955 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574745 5615 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:48:25.583955 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574748 5615 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:48:25.583955 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574750 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:48:25.583955 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574754 5615 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:48:25.583955 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574756 5615 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:48:25.583955 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574759 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:48:25.583955 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574762 5615 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:48:25.584443 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574764 5615 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:48:25.584443 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574767 5615 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:48:25.584443 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574769 5615 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:48:25.584443 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574773 5615 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:48:25.584443 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574775 5615 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:48:25.584443 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574778 5615 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:48:25.584443 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574781 5615 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:48:25.584443 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574784 5615 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:48:25.584443 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574787 5615 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:48:25.584443 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574789 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:48:25.584443 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574792 5615 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:48:25.584443 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574796 5615 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:48:25.584443 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574800 5615 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:48:25.584443 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574803 5615 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:48:25.584443 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574807 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:48:25.584443 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574810 5615 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:48:25.584443 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574813 5615 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:48:25.584443 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574817 5615 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:48:25.584443 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574820 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:48:25.584891 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574824 5615 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:48:25.584891 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574827 5615 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:48:25.584891 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574830 5615 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:48:25.584891 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574833 5615 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:48:25.584891 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574836 5615 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:48:25.584891 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574839 5615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:48:25.584891 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574842 5615 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:48:25.584891 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574844 5615 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:48:25.584891 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574847 5615 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:48:25.584891 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574850 5615 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:48:25.584891 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574853 5615 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:48:25.584891 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574856 5615 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:48:25.584891 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574858 5615 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:48:25.584891 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574861 5615 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:48:25.584891 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574864 5615 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:48:25.584891 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574866 5615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:48:25.584891 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574869 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:48:25.584891 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574872 5615 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:48:25.584891 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574875 5615 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:48:25.585365 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574877 5615 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:48:25.585365 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574880 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:48:25.585365 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574883 5615 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:48:25.585365 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574885 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:48:25.585365 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574888 5615 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:48:25.585365 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574891 5615 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:48:25.585365 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574893 5615 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:48:25.585365 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574898 5615 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:48:25.585365 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574902 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:48:25.585365 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574905 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:48:25.585365 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574908 5615 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:48:25.585365 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574911 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:48:25.585365 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574914 5615 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:48:25.585365 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574917 5615 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:48:25.585365 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574920 5615 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:48:25.585365 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574923 5615 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:48:25.585365 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574926 5615 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:48:25.585365 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574928 5615 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:48:25.585365 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574931 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:48:25.585816 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574934 5615 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:48:25.585816 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574936 5615 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:48:25.585816 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574939 5615 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:48:25.585816 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574942 5615 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:48:25.585816 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574944 5615 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:48:25.585816 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574947 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:48:25.585816 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574950 5615 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:48:25.585816 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574952 5615 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:48:25.585816 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574955 5615 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:48:25.585816 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574957 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:48:25.585816 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574960 5615 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:48:25.585816 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574963 5615 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:48:25.585816 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574965 5615 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:48:25.585816 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574969 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:48:25.585816 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574971 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:48:25.585816 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574974 5615 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:48:25.585816 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574976 5615 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:48:25.585816 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574979 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:48:25.585816 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574982 5615 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:48:25.585816 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574984 5615 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:48:25.586307 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574988 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:48:25.586307 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.574990 5615 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:48:25.586307 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.575019 5615 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:48:25.586307 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575181 5615 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:48:25.586307 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575186 5615 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:48:25.586307 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575189 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:48:25.586307 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575192 5615 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:48:25.586307 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575195 5615 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:48:25.586307 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575198 5615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:48:25.586307 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575201 5615 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:48:25.586307 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575203 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:48:25.586307 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575206 5615 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:48:25.586307 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575209 5615 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:48:25.586307 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575211 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:48:25.586307 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575214 5615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:48:25.586307 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575217 5615 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:48:25.586688 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575219 5615 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:48:25.586688 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575222 5615 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:48:25.586688 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575225 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:48:25.586688 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575228 5615 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:48:25.586688 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575232 5615 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:48:25.586688 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575235 5615 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:48:25.586688 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575238 5615 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:48:25.586688 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575241 5615 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:48:25.586688 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575244 5615 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:48:25.586688 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575247 5615 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:48:25.586688 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575250 5615 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:48:25.586688 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575252 5615 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:48:25.586688 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575255 5615 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:48:25.586688 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575258 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:48:25.586688 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575261 5615 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:48:25.586688 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575265 5615 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:48:25.586688 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575268 5615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:48:25.586688 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575272 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:48:25.586688 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575275 5615 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:48:25.587146 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575278 5615 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:48:25.587146 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575280 5615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:48:25.587146 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575283 5615 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:48:25.587146 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575286 5615 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:48:25.587146 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575289 5615 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:48:25.587146 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575292 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:48:25.587146 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575295 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:48:25.587146 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575298 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:48:25.587146 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575301 5615 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:48:25.587146 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575303 5615 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:48:25.587146 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575306 5615 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:48:25.587146 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575309 5615 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:48:25.587146 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575311 5615 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:48:25.587146 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575314 5615 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:48:25.587146 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575317 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:48:25.587146 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575319 5615 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:48:25.587146 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575322 5615 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:48:25.587146 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575325 5615 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:48:25.587146 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575328 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:48:25.587627 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575330 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:48:25.587627 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575333 5615 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:48:25.587627 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575336 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:48:25.587627 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575339 5615 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:48:25.587627 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575342 5615 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:48:25.587627 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575344 5615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:48:25.587627 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575347 5615 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:48:25.587627 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575350 5615 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:48:25.587627 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575353 5615 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:48:25.587627 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575356 5615 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:48:25.587627 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575359 5615 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:48:25.587627 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575361 5615 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:48:25.587627 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575364 5615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:48:25.587627 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575367 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:48:25.587627 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575369 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:48:25.587627 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575372 5615 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:48:25.587627 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575375 5615 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:48:25.587627 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575378 5615 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:48:25.587627 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575380 5615 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:48:25.587627 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575383 5615 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:48:25.588106 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575386 5615 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:48:25.588106 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575389 5615 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:48:25.588106 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575392 5615 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:48:25.588106 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575394 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:48:25.588106 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575397 5615 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:48:25.588106 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575400 5615 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:48:25.588106 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575403 5615 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:48:25.588106 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575405 5615 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:48:25.588106 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575408 5615 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:48:25.588106 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575410 5615 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:48:25.588106 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575413 5615 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:48:25.588106 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575417 5615 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:48:25.588106 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575419 5615 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:48:25.588106 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575422 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:48:25.588106 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575425 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:48:25.588106 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575428 5615 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:48:25.588106 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575430 5615 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:48:25.588106 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575433 5615 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:48:25.588106 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575436 5615 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:48:25.588553 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575439 5615 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:48:25.588553 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575441 5615 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:48:25.588553 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575444 5615 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:48:25.588553 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575447 5615 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:48:25.588553 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575450 5615 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:48:25.588553 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575453 5615 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:48:25.588553 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575455 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:48:25.588553 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575458 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:48:25.588553 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575460 5615 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:48:25.588553 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575463 5615 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:48:25.588553 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575466 5615 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:48:25.588553 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575468 5615 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:48:25.588553 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575471 5615 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:48:25.588553 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575474 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:48:25.588553 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575477 5615 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:48:25.588553 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575479 5615 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:48:25.588952 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.575503 5615 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:48:25.588952 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575536 5615 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:48:25.588952 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575539 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:48:25.588952 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575542 5615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:48:25.588952 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575545 5615 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:48:25.588952 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575548 5615 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:48:25.588952 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575550 5615 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:48:25.588952 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575553 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:48:25.588952 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575556 5615 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:48:25.588952 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575560 5615 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:48:25.588952 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575564 5615 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:48:25.588952 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575567 5615 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:48:25.588952 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575569 5615 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:48:25.588952 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575572 5615 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:48:25.588952 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575575 5615 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:48:25.588952 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575578 5615 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:48:25.589349 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575581 5615 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:48:25.589349 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575583 5615 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:48:25.589349 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575586 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:48:25.589349 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575590 5615 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:48:25.589349 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575594 5615 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:48:25.589349 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575597 5615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:48:25.589349 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575600 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:48:25.589349 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575602 5615 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:48:25.589349 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575605 5615 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:48:25.589349 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575608 5615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:48:25.589349 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575611 5615 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:48:25.589349 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575613 5615 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:48:25.589349 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575616 5615 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:48:25.589349 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575619 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:48:25.589349 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575621 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:48:25.589349 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575624 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:48:25.589349 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575628 5615 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:48:25.589349 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575630 5615 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:48:25.589349 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575633 5615 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:48:25.589803 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575636 5615 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:48:25.589803 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575638 5615 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:48:25.589803 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575641 5615 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:48:25.589803 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575643 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:48:25.589803 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575646 5615 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:48:25.589803 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575649 5615 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:48:25.589803 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575651 5615 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:48:25.589803 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575656 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:48:25.589803 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575659 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:48:25.589803 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575662 5615 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:48:25.589803 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575665 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:48:25.589803 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575667 5615 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:48:25.589803 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575670 5615 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:48:25.589803 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575673 5615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:48:25.589803 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575675 5615 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:48:25.589803 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575678 5615 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:48:25.589803 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575681 5615 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:48:25.589803 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575683 5615 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:48:25.589803 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575686 5615 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:48:25.589803 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575689 5615 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:48:25.590446 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575691 5615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:48:25.590446 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575694 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:48:25.590446 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575696 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:48:25.590446 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575699 5615 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:48:25.590446 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575701 5615 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:48:25.590446 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575704 5615 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:48:25.590446 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575707 5615 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:48:25.590446 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575710 5615 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:48:25.590446 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575713 5615 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:48:25.590446 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575717 5615 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:48:25.590446 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575720 5615 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:48:25.590446 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575722 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:48:25.590446 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575726 5615 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:48:25.590446 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575728 5615 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:48:25.590446 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575731 5615 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:48:25.590446 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575733 5615 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:48:25.590446 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575736 5615 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:48:25.590446 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575739 5615 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:48:25.590446 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575741 5615 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:48:25.590895 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575745 5615 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:48:25.590895 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575748 5615 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:48:25.590895 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575751 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:48:25.590895 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575753 5615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:48:25.590895 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575756 5615 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:48:25.590895 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575759 5615 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:48:25.590895 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575761 5615 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:48:25.590895 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575764 5615 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:48:25.590895 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575767 5615 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:48:25.590895 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575769 5615 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:48:25.590895 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575772 5615 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:48:25.590895 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575775 5615 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:48:25.590895 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575778 5615 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:48:25.590895 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575781 5615 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:48:25.590895 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575784 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:48:25.590895 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575786 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:48:25.590895 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575789 5615 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:48:25.590895 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575794 5615 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:48:25.590895 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575797 5615 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:48:25.590895 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575799 5615 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:48:25.591388 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575802 5615 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:48:25.591388 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575805 5615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:48:25.591388 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575808 5615 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:48:25.591388 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575811 5615 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:48:25.591388 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575814 5615 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:48:25.591388 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575816 5615 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:48:25.591388 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575819 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:48:25.591388 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575822 5615 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:48:25.591388 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575824 5615 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:48:25.591388 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575827 5615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:48:25.591388 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575830 5615 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:48:25.591388 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575832 5615 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:48:25.591388 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:25.575836 5615 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:48:25.591388 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.575898 5615 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 20:48:25.591388 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.576961 5615 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 20:48:25.591388 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.577022 5615 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 20:48:25.591388 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.577218 5615 server.go:1014] "Starting client certificate rotation" Nov 03 20:48:25.591388 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.577351 5615 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 20:48:25.591842 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.577395 5615 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 20:42:14 +0000 UTC" deadline="2028-04-21 12:21:55.228052893 +0000 UTC" Nov 03 20:48:25.591842 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.577420 5615 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12831h33m29.650634392s" Nov 03 20:48:25.591842 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.577654 5615 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:48:25.591842 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.578874 5615 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:48:25.591842 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.580130 5615 log.go:25] "Validated CRI v1 runtime API" Nov 03 20:48:25.591842 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.581997 5615 log.go:25] "Validated CRI v1 image API" Nov 03 20:48:25.591842 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.582678 5615 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 20:48:25.591842 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.584864 5615 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 99e37395-ebd1-47ab-b414-5eda35e76ff4:/dev/nvme0n1p3 a0ca9470-af9b-4137-8afa-e76e68339521:/dev/nvme0n1p4] Nov 03 20:48:25.592062 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.584884 5615 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0308d0c818cb3fa16f8a663af38380509a71f2b74e894937f2e4961830a36dc2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0308d0c818cb3fa16f8a663af38380509a71f2b74e894937f2e4961830a36dc2/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0d1c0d124ccb983ea54849dfaf3549fc40b31296c38641a01591988123f40fc5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0d1c0d124ccb983ea54849dfaf3549fc40b31296c38641a01591988123f40fc5/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/14a9ff28a6078cfcb41c163e90a0d9ba2abb7f39cf9fe994bc2a0cbc5e5b6cb0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/14a9ff28a6078cfcb41c163e90a0d9ba2abb7f39cf9fe994bc2a0cbc5e5b6cb0/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/30c9d23cea06d2e2ee7e26b675a4eadcd7b354fc941609a1dcae96f319a95d40/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/30c9d23cea06d2e2ee7e26b675a4eadcd7b354fc941609a1dcae96f319a95d40/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46ee947a9f7dbd243e535b5be14ff904040c24708f75e786f36c9ea6fe1b31e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46ee947a9f7dbd243e535b5be14ff904040c24708f75e786f36c9ea6fe1b31e2/userdata/shm major:0 minor:337 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/546bdb5314f9cefee256c637fee36db01e128465d8deb23c89c9021544635501/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/546bdb5314f9cefee256c637fee36db01e128465d8deb23c89c9021544635501/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6290d19f6ae8e83908c9bafdc5c1ea7f8b285d15954b85cbf70a2f1de6065d98/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6290d19f6ae8e83908c9bafdc5c1ea7f8b285d15954b85cbf70a2f1de6065d98/userdata/shm major:0 minor:423 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/740a6d0a1a116d13d11958b6f022a8a647d699b659a1b9b83ee9a8acae3b8e22/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/740a6d0a1a116d13d11958b6f022a8a647d699b659a1b9b83ee9a8acae3b8e22/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8511e8faa04dfe9c6a804c5e3b4990480722ceaed7d8012f4667c5410dd8332a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8511e8faa04dfe9c6a804c5e3b4990480722ceaed7d8012f4667c5410dd8332a/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c6bb37beee1291cde5d34c9079e670822933507e30b67380f8d3ff64a3f38ee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c6bb37beee1291cde5d34c9079e670822933507e30b67380f8d3ff64a3f38ee/userdata/shm major:0 minor:422 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8fb0cf8cc04cb3dc87b3cab3765ae19afff3daf13b90b98b1286baca880ba23b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8fb0cf8cc04cb3dc87b3cab3765ae19afff3daf13b90b98b1286baca880ba23b/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/90cc044bb1c64adc17302b5dbc017fc1bb157884b0ff69d27f0574d5fc61d689/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/90cc044bb1c64adc17302b5dbc017fc1bb157884b0ff69d27f0574d5fc61d689/userdata/shm major:0 minor:426 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/92de2ebb594e68d825289f2412232e1a91a62be2e243281001e49b04488d85ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/92de2ebb594e68d825289f2412232e1a91a62be2e243281001e49b04488d85ad/userdata/shm major:0 minor:254 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/95a33df2d7282b2989b0a95d55a9d31eb9be8940fab157789f5a8e3fe0414f91/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/95a33df2d7282b2989b0a95d55a9d31eb9be8940fab157789f5a8e3fe0414f91/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f1dde26a34e0e5d20517e130b74b6c274b1d6a431637f4a41083294a50b2e2a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f1dde26a34e0e5d20517e130b74b6c274b1d6a431637f4a41083294a50b2e2a/userdata/shm major:0 minor:339 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a1513668b128288d80b165d2599ebd99184addb467d8ed53c0f247d9e02a4af6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a1513668b128288d80b165d2599ebd99184addb467d8ed53c0f247d9e02a4af6/userdata/shm major:0 minor:258 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a2629873f88b6fb29b09dd4a207c332ea166c13a739da8c750cf06f33bee101e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a2629873f88b6fb29b09dd4a207c332ea166c13a739da8c750cf06f33bee101e/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a2a172ce3de787560a04bafaebe785f49aebf30c6a2179418bd1dcef7cea3328/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a2a172ce3de787560a04bafaebe785f49aebf30c6a2179418bd1dcef7cea3328/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bda995bdc42eaca76d1359b111bfb323e63e37da83761d452d97498210ee75c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bda995bdc42eaca76d1359b111bfb323e63e37da83761d452d97498210ee75c5/userdata/shm major:0 minor:262 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2fdc7bebf927bd4cb7bd5a1e9fc6710f63867dfc7a957648e812082424e6b28/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2fdc7bebf927bd4cb7bd5a1e9fc6710f63867dfc7a957648e812082424e6b28/userdata/shm major:0 minor:353 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e666b74d740ff3bc6f845fa19ccf562c1e0ee6a6f6475572c685ac7d72b6fbb3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e666b74d740ff3bc6f845fa19ccf562c1e0ee6a6f6475572c685ac7d72b6fbb3/userdata/shm major:0 minor:327 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eed08506c695a4b7e75761bc37391ce9891831f7f93b47b1d781fd9c62ef6d8f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eed08506c695a4b7e75761bc37391ce9891831f7f93b47b1d781fd9c62ef6d8f/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f60e0f8c9bd1538eeca368800ccd9da47cda90699ebc3e92d80a6a7b3ec38513/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f60e0f8c9bd1538eeca368800ccd9da47cda90699ebc3e92d80a6a7b3ec38513/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f6a2381d6d2ec12b671bc70f4b440c2c384ebc426ede6d9f412f720f3d3f1241/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f6a2381d6d2ec12b671bc70f4b440c2c384ebc426ede6d9f412f720f3d3f1241/userdata/shm major:0 minor:316 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/026b5dc8-bcff-4afc-a784-baf95c1cec27/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/026b5dc8-bcff-4afc-a784-baf95c1cec27/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/026b5dc8-bcff-4afc-a784-baf95c1cec27/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/026b5dc8-bcff-4afc-a784-baf95c1cec27/volumes/kubernetes.io~empty-dir/tmp major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/026b5dc8-bcff-4afc-a784-baf95c1cec27/volumes/kubernetes.io~projected/kube-api-access-p6lpt:{mountpoint:/var/lib/kubelet/pods/026b5dc8-bcff-4afc-a784-baf95c1cec27/volumes/kubernetes.io~projected/kube-api-access-p6lpt major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0327c758-d1f5-481d-9cf2-96e4b5f27872/volumes/kubernetes.io~projected/kube-api-access-p7rds:{mountpoint:/var/lib/kubelet/pods/0327c758-d1f5-481d-9cf2-96e4b5f27872/volumes/kubernetes.io~projected/kube-api-access-p7rds major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0327c758-d1f5-481d-9cf2-96e4b5f27872/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/0327c758-d1f5-481d-9cf2-96e4b5f27872/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0327c758-d1f5-481d-9cf2-96e4b5f27872/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/0327c758-d1f5-481d-9cf2-96e4b5f27872/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0327c758-d1f5-481d-9cf2-96e4b5f27872/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/0327c758-d1f5-481d-9cf2-96e4b5f27872/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0327c758-d1f5-481d-9cf2-96e4b5f27872/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/0327c758-d1f5-481d-9cf2-96e4b5f27872/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05683773-8c40-4d7c-9cc9-ee9d71b25b23/volumes/kubernetes.io~projected/kube-api-access-wc6n7:{mountpoint:/var/lib/kubelet/pods/05683773-8c40-4d7c-9cc9-ee9d71b25b23/volumes/kubernetes.io~projected/kube-api-access-wc6n7 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05683773-8c40-4d7c-9cc9-ee9d71b25b23/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/05683773-8c40-4d7c-9cc9-ee9d71b25b23/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15a5eea7-15b5-45a6-838b-35914cb59be1/volumes/kubernetes.io~projected/kube-api-access-x6x9d:{mountpoint:/var/lib/kubelet/pods/15a5eea7-15b5-45a6-838b-35914cb59be1/volumes/kubernetes.io~projected/kube-api-access-x6x9d major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15a5eea7-15b5-45a6-838b-35914cb59be1/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/15a5eea7-15b5-45a6-838b-35914cb59be1/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15a5eea7-15b5-45a6-838b-35914cb59be1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/15a5eea7-15b5-45a6-838b-35914cb59be1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15a5eea7-15b5-45a6-838b-35914cb59be1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/15a5eea7-15b5-45a6-838b-35914cb59be1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15a5eea7-15b5-45a6-838b-35914cb59be1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/15a5eea7-15b5-45a6-838b-35914cb59be1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15a5eea7-15b5-45a6-838b-35914cb59be1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/15a5eea7-15b5-45a6-838b-35914cb59be1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15a5eea7-15b5-45a6-838b-35914cb59be1/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/15a5eea7-15b5-45a6-838b-35914cb59be1/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/195d22be-670c-4866-a303-1333085f8a67/volumes/kubernetes.io~projected/kube-api-access-vw4cq:{mountpoint:/var/lib/kubelet/pods/195d22be-670c-4866-a303-1333085f8a67/volumes/kubernetes.io~projected/kube-api-access-vw4cq major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/195d22be-670c-4866-a303-1333085f8a67/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/195d22be-670c-4866-a303-1333085f8a67/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/195d22be-670c-4866-a303-1333085f8a67/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/195d22be-670c-4866-a303-1333085f8a67/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a695777-f42a-4cb6-8a31-c07c8daf321a/volumes/kubernetes.io~projected/kube-api-access-lhkvl:{mountpoint:/var/lib/kubelet/pods/3a695777-f42a-4cb6-8a31-c07c8daf321a/volumes/kubernetes.io~projected/kube-api-access-lhkvl major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a695777-f42a-4cb6-8a31-c07c8daf321a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/3a695777-f42a-4cb6-8a31-c07c8daf321a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:250 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a695777-f42a-4cb6-8a31-c07c8daf321a/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/3a695777-f42a-4cb6-8a31-c07c8daf321a/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/712fcecb-4757-4dd0-8f96-084b1e6ee903/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/712fcecb-4757-4dd0-8f96-084b1e6ee903/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:420 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9050192a-0ec2-45b9-8b2c-2beb0501b09b/volumes/kubernetes.io~projected/kube-api-access-tv6zc:{mountpoint:/var/lib/kubelet/pods/9050192a-0ec2-45b9-8b2c-2beb0501b09b/volumes/kubernetes.io~projected/kube-api-access-tv6zc major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81/volumes/kubernetes.io~projected/kube-api-access-j5mgg:{mountpoint:/var/lib/kubelet/pods/955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81/volumes/kubernetes.io~projected/kube-api-access-j5mgg major:0 minor:421 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2dc043c-52c0-4da0-8370-ed975ebb987f/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/a2dc043c-52c0-4da0-8370-ed975ebb987f/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2dc043c-52c0-4da0-8370-ed975ebb987f/volumes/kubernetes.io~projected/kube-api-access-42v78:{mountpoint:/var/lib/kubelet/pods/a2dc043c-52c0-4da0-8370-ed975ebb987f/volumes/kubernetes.io~projected/kube-api-access-42v78 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2dc043c-52c0-4da0-8370-ed975ebb987f/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/a2dc043c-52c0-4da0-8370-ed975ebb987f/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3e654a3-7d76-4793-9627-6334ba6db6d9/volumes/kubernetes.io~projected/kube-api-access-prrnl:{mountpoint:/var/lib/kubelet/pods/a3e654a3-7d76-4793-9627-6334ba6db6d9/volumes/kubernetes.io~projected/kube-api-access-prrnl major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a882e61f-2b83-4ff4-8370-cba401a6d9c8/volumes/kubernetes.io~projected/kube-api-access-6ttzr:{mountpoint:/var/lib/kubelet/pods/a882e61f-2b83-4ff4-8370-cba401a6d9c8/volumes/kubernetes.io~projected/kube-api-access-6ttzr major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb/volumes/kubernetes.io~projected/kube-api-access-cbctt:{mountpoint:/var/lib/kubelet/pods/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb/volumes/kubernetes.io~projected/kube-api-access-cbctt major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bee40e70-1cd3-4b89-8808-3abf1dcfbee9/volumes/kubernetes.io~projected/kube-api-access-rbhtz:{mountpoint:/var/lib/kubelet/pods/bee40e70-1cd3-4b89-8808-3abf1dcfbee9/volumes/kubernetes.io~projected/kube-api-access-rbhtz major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bee40e70-1cd3-4b89-8808-3abf1dcfbee9/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/bee40e70-1cd3-4b89-8808-3abf1dcfbee9/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfca9b06-e0d2-4d0f-a75c-7827485cb42d/volumes/kubernetes.io~projected/kube-api-access-tf5tz:{mountpoint:/var/lib/kubelet/pods/bfca9b06-e0d2-4d0f-a75c-7827485cb42d/volumes/kubernetes.io~projected/kube-api-access-tf5tz major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c/volumes/kubernetes.io~projected/kube-api-access-fbvc2:{mountpoint:/var/lib/kubelet/pods/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c/volumes/kubernetes.io~projected/kube-api-access-fbvc2 major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:249 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c/volumes/kubernetes.io~projected/kube-api-access-dh5qw:{mountpoint:/var/lib/kubelet/pods/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c/volumes/kubernetes.io~projected/kube-api-access-dh5qw major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c/volumes/kubernetes.io~secret/metrics-certs major:0 minor:416 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c77a4d5b-a0d3-468b-aa66-7d607794aba1/volumes/kubernetes.io~projected/kube-api-access-9pgk2:{mountpoint:/var/lib/kubelet/pods/c77a4d5b-a0d3-468b-aa66-7d607794aba1/volumes/kubernetes.io~projected/kube-api-access-9pgk2 major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c77a4d5b-a0d3-468b-aa66-7d607794aba1/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/c77a4d5b-a0d3-468b-aa66-7d607794aba1/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c77a4d5b-a0d3-468b-aa66-7d607794aba1/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/c77a4d5b-a0d3-468b-aa66-7d607794aba1/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cba96b6a-ccc8-4c1c-b371-1bb5ece29afc/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/cba96b6a-ccc8-4c1c-b371-1bb5ece29afc/volumes/kubernetes.io~secret/agent-certs major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e64dc44d-5470-4abb-8155-fdf7f24c253b/volumes/kubernetes.io~projected/kube-api-access-zzx44:{mountpoint:/var/lib/kubelet/pods/e64dc44d-5470-4abb-8155-fdf7f24c253b/volumes/kubernetes.io~projected/kube-api-access-zzx44 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e64dc44d-5470-4abb-8155-fdf7f24c253b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/e64dc44d-5470-4abb-8155-fdf7f24c253b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eaf3e4d3-7019-42e5-9e1e-2927b7f22304/volumes/kubernetes.io~projected/kube-api-access-nkxvg:{mountpoint:/var/lib/kubelet/pods/eaf3e4d3-7019-42e5-9e1e-2927b7f22304/volumes/kubernetes.io~projected/kube-api-access-nkxvg major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ebf82283-d275-4640-bc44-5a461e4f9f9d/volumes/kubernetes.io~projected/kube-api-access-pbsdk:{mountpoint:/var/lib/kubelet/pods/ebf82283-d275-4640-bc44-5a461e4f9f9d/volumes/kubernetes.io~projected/kube-api-access-pbsdk major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ebf82283-d275-4640-bc44-5a461e4f9f9d/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/ebf82283-d275-4640-bc44-5a461e4f9f9d/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ebf82283-d275-4640-bc44-5a461e4f9f9d/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/ebf82283-d275-4640-bc44-5a461e4f9f9d/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ebf82283-d275-4640-bc44-5a461e4f9f9d/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/ebf82283-d275-4640-bc44-5a461e4f9f9d/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fac5da49-6649-4779-9529-089804436313/volumes/kubernetes.io~projected/kube-api-access-k85jh:{mountpoint:/var/lib/kubelet/pods/fac5da49-6649-4779-9529-089804436313/volumes/kubernetes.io~projected/kube-api-access-k85jh major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fac5da49-6649-4779-9529-089804436313/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/fac5da49-6649-4779-9529-089804436313/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/fed50579f9e088289979e330d74ee3ab0b5b4c602312c0b58c600e1ac5fc3fef/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/f7f3c14dfa3d817a6b4bc7b12465a0ea11eb37c3663ce59db268042f9878d8e7/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/4288f9fd6ea3e9cdbfc2e757087c0728956732ef8d0ae252d569e722a3d60039/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/127287db2dd9504e40215e9580950d123c262d04ea85cc57a1c2157a293a6cc1/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/87d2ad1257d146b05db06f060d9e672e4703fe651388ca0f6b54f6eb223ae5db/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/64746ba5d160055846a3f03d2a4285d9221a0fab50835076609939614c220ddb/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/eed747c3affe5b5b0feacec30054996ca1f703fb0f37b454918b4e5db204a494/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/238e4d61937df0e7a7e223518afa4368dca7b02427f940f372ecec95b536aa8a/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/3d8bc0dbe102f18fb6fe0a7358c052553b58119054b699a8ff995219d93333e9/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/42154a6311a0354cee518b122727bec57925cdac94e5c45a7712f983c6c3394f/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/d1008e39bd3f2f885cefae62aacd17e40857007115819dd842e8c62d5d0a34c7/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/a820aa0fdebe1482f85a4b768067946f324db150bf3ee7aff588ae79a88b1874/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/d0ae824f1c5521d643168d88b8f7ac965cc0ab82527c052b38a6a4abd59651ad/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/9cbcabc12059d406b2827568d56a2d7670beb120943ba5bc0ec23c5fdf09305f/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/83ff93686630bed96d112b4b500a1d14c4ac81c57a7fcdb6af2bcd0294252369/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/b597e61291bfd3193c1e70e89c5ef8ca776585e41c85806e8bc18da81fc8d2e5/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/d41991939dc65a8b98b210bf6b679a34b385b2e3010d506edb34d4a2f426e151/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/0812f29c370512dcad3e6d1968e209d30b801ee54697f9b8be2b5017c6f3bd90/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/e686a20ed8fb7e6105825d31e7d83fe9958ee6af8891448af1537e8542e7f186/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/36325144b4b89cc05d1520c96fa683db9dda4cc3a4c323b97c99356f27979ac9/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/b12897d5441a59846c27a8b1026a6587b23f4a86dd7ce5e47fd1777bbd6db774/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/79095f8e9985075e47366493a5ff2a4a91d3b2c5c9cdb8e3996411b6f95c9683/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/c0302de89e9f37abe7d610c467c4b220cc96565af1fd401fb2a6c27e2f894498/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/b06f6da1555bb562ce53546557f4867604f9a538bc4a75887629623f3d9e8055/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/38aa283d0b104573d14e08f10dfeb656cf34d5ed0007aafbe680146bafb93094/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/9d99f10ecf293eff37942f84a6f68ae70f86714a9df92c49aacd729da49d2b9e/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/e4e982ed83d1e4276d547a9293f151fd36714b0587fbf0795eead065770bf2f5/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/8468942c43deb92e21e67fc8f186f3764ae15621031d1b694686ec7946ce1041/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/c310f47e10c77f2de61a3ea758bf2784fc87f89574a33b9336daf1d90fad8c47/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/0c0b2c1d75d13285ee785f902a4b344e0172026266e2bbf0157d6ba1d5a9aeaf/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/617a1be57731fdd3f0b0c5e1dd6e2307da8445bbac12c77e7bc4fac7d3185290/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/3aa7ebeb26a90b6bf64702c20e808fe44f82dba3ea2903deff6f04f5f2cb1d13/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/d44bda373460b1bf99523d15d99bd54a6ccc3c0d3a117e745db1a108eb72db32/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/c89c1db9e047081a91250fae440d6445a3a1d8b18f5f663d49c8fad4769cadf8/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/d3a926b48ffef55f884d1e59851c3da649bb5e80543535c4e2ee97e51c459f7a/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/d19ed02df3ee14ca7c80086f912d0c13e80c37ca439aa975d80d35c2acf4fd3f/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/170f659e965186e29fe48c4da8d689fcdb1b6ea8cd066a97d9968771a1600714/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/5b524f289db10658e81f4372cfa1ff9dd39674c36625faaf301f269cb95addc9/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/b93d68624b37b0c1b680d576e4c5e094c61fd66cb36204ef339ee879535c113f/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/bafc529e8715260cd04fe55ff156da9c04dd6cc91b448a6bed647713b8c502e8/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/55992b5ff477bb136ec9ddd236b910e90301b6fd5c8d9b9bb8287280ec116b56/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/9a3969315a43ba746f7654c3d09beb0c769ca29646663144355cb044ada6282e/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/2eaaff1cefbdd5fd3b9e7c606824a41aef8ded3b7546e39583ffa36f53ef7e42/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/5f76966859221579908f0bcf1cc4fc3c59352a3afbc692d0d6847afc8a360a66/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/9e7a4caa0fcc04e0119cdaef859e771923302ae8fa3c46a68b0577a0dce85975/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/78d7ce5dc4c10ba1929da066bbf0a4c92460b270c9b212f9a1023b2a3604d6a2/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/0517d3d7d6d59e10c984859e3c3da75782c7d45d04ef96b812c7a4470ef2408f/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/2bc69a4e94ba9009b9d3d89a7554e45aa0c4665ab82af024ee52f3b874667324/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/a47b9de50a17e5ea47846aab2b1ba15fa019a05cdfe434a8f395f478de7da3be/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/c55796908581c93ea846cb0093467f11e7d27693517fa758d3cfd4965ff70539/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/e19ec77cb41b311373ea5177ef8a4d9980ed68164276e3915a1dd96cb432cbd6/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/c0058d643df16e2c3d79dc7a39851f33ef96289c9869d70147e534416e053d5b/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/76723747fcfd7db457c4c4e3485fd25502a60f9adde440c8e1477dc81c408ec3/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/80dc9a4156a65aa22e12b5dfa2a3b9e773c13b412bb52a9c50cef4d52434231a/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/5ec9b482199dc14d8e45e68b50cad2fd5e403ce278a6fde1f0270f42df6d2f9b/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/72bb61c058df4359c4086ef01beb15cb049d0b7e0db89067df60e94537fd1b2f/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-405:{mountpoint:/var/lib/containers/storage/overlay/bd3da0463345bdc7b4913402041851132223d1f6c18d09f3b90ab66574dd6b59/merged major:0 minor:405 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/0d29192d8d7f0b8647a6083ead2a908a0c2b0007e634143eaa4aef70ab0d261c/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-428:{mountpoint:/var/lib/containers/storage/overlay/7d41c6d38811fe1842471c10b5fcd1447b640e7c80b2c742ced6eda353758db4/merged major:0 minor:428 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/89607836310a8eb9fcfe65117ad4924d2eac1784c881d5a62ea70a908d3e12cb/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-432:{mountpoint:/var/lib/containers/storage/overlay/baa06f832f5c64154dd831a9e70719fc1ac92447e763e0daf6edec8aa41fff6a/merged major:0 minor:432 fsType:overlay blockSize:0} overlay_0-434:{mountpoint:/var/lib/containers/storage/overlay/538f24e0d9ac1e01c256d5efa6effbd60dcc887315a8fa4c5343058057e6ba8f/merged major:0 minor:434 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/c37d6d7f17cd1fca6f190b864dc0bfde7c57616b17e90cbc82ffa48c765dcf14/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/0a923471996ed53d8d115886004700a424918b2e4cf2add42a75281aeb3b4b6e/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-447:{mountpoint:/var/lib/containers/storage/overlay/aff1bbd09810b4ef89b4ec324543fcd162578b8ed64f73d3e166107ddcfb1f37/merged major:0 minor:447 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/42fd3c42b7801ebbd8cda5ad9742065e4670eaec5b1b4ec910a29c565828dc42/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/41d8848da4b4bb31e02f477518efda5d5215914e1b37a9549ac877de2ec2a9f7/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/7ef345d223518cc2a022aff523cf4b1bc2e0e05c026495a3d3050df4cebac54d/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/56200edcc33e3448a8995800deceeddb2ad0131eb6ce080df5bebf9fc974ce70/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/bcb64363f0d22cb350e38b3cd2563f726c292b8e336b6c1b0c954395dcc0b334/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/c228a5af63af1d9219ed05206d430694ace5f41aba33139bae2ae09e50dbd64e/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/eb220f4e0950eef1f18e89de035c433076a22fbdb64c6fbb90166ccfcf77fbfc/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/6260596e62f544afd9551accbf07322279a7e15bfa2f37637c08830bb7733b28/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/1707eac7045c0249abeaebf2fa9858b341314543c7df2044c131c7e62d90a3fc/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/41567874d56ff6e0ed76bfb8e6da5c2454e23717abe8f00db8cc9b23260a8b0a/merged major:0 minor:98 fsType:overlay blockSize:0}] Nov 03 20:48:25.595676 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.595285 5615 manager.go:217] Machine: {Timestamp:2026-11-03 20:48:25.594800255 +0000 UTC m=+0.062578969 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098983 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e399285b3e2332cde22c26326417d SystemUUID:ec2e3992-85b3-e233-2cde-22c26326417d BootID:f8013594-09f1-4ea0-9b43-075a377e4ef2 Filesystems:[{Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/546bdb5314f9cefee256c637fee36db01e128465d8deb23c89c9021544635501/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c77a4d5b-a0d3-468b-aa66-7d607794aba1/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:246 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/92de2ebb594e68d825289f2412232e1a91a62be2e243281001e49b04488d85ad/userdata/shm DeviceMajor:0 DeviceMinor:254 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0327c758-d1f5-481d-9cf2-96e4b5f27872/volumes/kubernetes.io~projected/kube-api-access-p7rds DeviceMajor:0 DeviceMinor:349 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-405 DeviceMajor:0 DeviceMinor:405 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2fdc7bebf927bd4cb7bd5a1e9fc6710f63867dfc7a957648e812082424e6b28/userdata/shm DeviceMajor:0 DeviceMinor:353 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/a882e61f-2b83-4ff4-8370-cba401a6d9c8/volumes/kubernetes.io~projected/kube-api-access-6ttzr DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/15a5eea7-15b5-45a6-838b-35914cb59be1/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:296 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/14a9ff28a6078cfcb41c163e90a0d9ba2abb7f39cf9fe994bc2a0cbc5e5b6cb0/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e64dc44d-5470-4abb-8155-fdf7f24c253b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bda995bdc42eaca76d1359b111bfb323e63e37da83761d452d97498210ee75c5/userdata/shm DeviceMajor:0 DeviceMinor:262 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c77a4d5b-a0d3-468b-aa66-7d607794aba1/volumes/kubernetes.io~projected/kube-api-access-9pgk2 DeviceMajor:0 DeviceMinor:247 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/15a5eea7-15b5-45a6-838b-35914cb59be1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:297 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/a2dc043c-52c0-4da0-8370-ed975ebb987f/volumes/kubernetes.io~projected/kube-api-access-42v78 DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f1dde26a34e0e5d20517e130b74b6c274b1d6a431637f4a41083294a50b2e2a/userdata/shm DeviceMajor:0 DeviceMinor:339 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2dc043c-52c0-4da0-8370-ed975ebb987f/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/3a695777-f42a-4cb6-8a31-c07c8daf321a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:250 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e666b74d740ff3bc6f845fa19ccf562c1e0ee6a6f6475572c685ac7d72b6fbb3/userdata/shm DeviceMajor:0 DeviceMinor:327 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:416 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/026b5dc8-bcff-4afc-a784-baf95c1cec27/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e64dc44d-5470-4abb-8155-fdf7f24c253b/volumes/kubernetes.io~projected/kube-api-access-zzx44 DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f6a2381d6d2ec12b671bc70f4b440c2c384ebc426ede6d9f412f720f3d3f1241/userdata/shm DeviceMajor:0 DeviceMinor:316 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a695777-f42a-4cb6-8a31-c07c8daf321a/volumes/kubernetes.io~projected/kube-api-access-lhkvl DeviceMajor:0 DeviceMinor:252 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2dc043c-52c0-4da0-8370-ed975ebb987f/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:251 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c/volumes/kubernetes.io~projected/kube-api-access-fbvc2 DeviceMajor:0 DeviceMinor:253 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/195d22be-670c-4866-a303-1333085f8a67/volumes/kubernetes.io~projected/kube-api-access-vw4cq DeviceMajor:0 DeviceMinor:326 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0327c758-d1f5-481d-9cf2-96e4b5f27872/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:347 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ebf82283-d275-4640-bc44-5a461e4f9f9d/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:334 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8511e8faa04dfe9c6a804c5e3b4990480722ceaed7d8012f4667c5410dd8332a/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a2a172ce3de787560a04bafaebe785f49aebf30c6a2179418bd1dcef7cea3328/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/195d22be-670c-4866-a303-1333085f8a67/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:319 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81/volumes/kubernetes.io~projected/kube-api-access-j5mgg DeviceMajor:0 DeviceMinor:421 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb/volumes/kubernetes.io~projected/kube-api-access-cbctt DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/195d22be-670c-4866-a303-1333085f8a67/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:323 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/740a6d0a1a116d13d11958b6f022a8a647d699b659a1b9b83ee9a8acae3b8e22/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fac5da49-6649-4779-9529-089804436313/volumes/kubernetes.io~projected/kube-api-access-k85jh DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a1513668b128288d80b165d2599ebd99184addb467d8ed53c0f247d9e02a4af6/userdata/shm DeviceMajor:0 DeviceMinor:258 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/026b5dc8-bcff-4afc-a784-baf95c1cec27/volumes/kubernetes.io~projected/kube-api-access-p6lpt DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/05683773-8c40-4d7c-9cc9-ee9d71b25b23/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0308d0c818cb3fa16f8a663af38380509a71f2b74e894937f2e4961830a36dc2/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/15a5eea7-15b5-45a6-838b-35914cb59be1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:295 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/15a5eea7-15b5-45a6-838b-35914cb59be1/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:303 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/90cc044bb1c64adc17302b5dbc017fc1bb157884b0ff69d27f0574d5fc61d689/userdata/shm DeviceMajor:0 DeviceMinor:426 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c/volumes/kubernetes.io~projected/kube-api-access-dh5qw DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/95a33df2d7282b2989b0a95d55a9d31eb9be8940fab157789f5a8e3fe0414f91/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fac5da49-6649-4779-9529-089804436313/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/15a5eea7-15b5-45a6-838b-35914cb59be1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:298 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0327c758-d1f5-481d-9cf2-96e4b5f27872/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:345 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/712fcecb-4757-4dd0-8f96-084b1e6ee903/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:420 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-432 DeviceMajor:0 DeviceMinor:432 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eaf3e4d3-7019-42e5-9e1e-2927b7f22304/volumes/kubernetes.io~projected/kube-api-access-nkxvg DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a695777-f42a-4cb6-8a31-c07c8daf321a/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:248 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46ee947a9f7dbd243e535b5be14ff904040c24708f75e786f36c9ea6fe1b31e2/userdata/shm DeviceMajor:0 DeviceMinor:337 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bfca9b06-e0d2-4d0f-a75c-7827485cb42d/volumes/kubernetes.io~projected/kube-api-access-tf5tz DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cba96b6a-ccc8-4c1c-b371-1bb5ece29afc/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a3e654a3-7d76-4793-9627-6334ba6db6d9/volumes/kubernetes.io~projected/kube-api-access-prrnl DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/15a5eea7-15b5-45a6-838b-35914cb59be1/volumes/kubernetes.io~projected/kube-api-access-x6x9d DeviceMajor:0 DeviceMinor:294 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ebf82283-d275-4640-bc44-5a461e4f9f9d/volumes/kubernetes.io~projected/kube-api-access-pbsdk DeviceMajor:0 DeviceMinor:336 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0327c758-d1f5-481d-9cf2-96e4b5f27872/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:346 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a2629873f88b6fb29b09dd4a207c332ea166c13a739da8c750cf06f33bee101e/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c77a4d5b-a0d3-468b-aa66-7d607794aba1/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:241 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:249 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0327c758-d1f5-481d-9cf2-96e4b5f27872/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:348 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-447 DeviceMajor:0 DeviceMinor:447 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0d1c0d124ccb983ea54849dfaf3549fc40b31296c38641a01591988123f40fc5/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bee40e70-1cd3-4b89-8808-3abf1dcfbee9/volumes/kubernetes.io~projected/kube-api-access-rbhtz DeviceMajor:0 DeviceMinor:332 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-434 DeviceMajor:0 DeviceMinor:434 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f60e0f8c9bd1538eeca368800ccd9da47cda90699ebc3e92d80a6a7b3ec38513/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6290d19f6ae8e83908c9bafdc5c1ea7f8b285d15954b85cbf70a2f1de6065d98/userdata/shm DeviceMajor:0 DeviceMinor:423 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-428 DeviceMajor:0 DeviceMinor:428 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8fb0cf8cc04cb3dc87b3cab3765ae19afff3daf13b90b98b1286baca880ba23b/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eed08506c695a4b7e75761bc37391ce9891831f7f93b47b1d781fd9c62ef6d8f/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/15a5eea7-15b5-45a6-838b-35914cb59be1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:302 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c6bb37beee1291cde5d34c9079e670822933507e30b67380f8d3ff64a3f38ee/userdata/shm DeviceMajor:0 DeviceMinor:422 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/026b5dc8-bcff-4afc-a784-baf95c1cec27/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/05683773-8c40-4d7c-9cc9-ee9d71b25b23/volumes/kubernetes.io~projected/kube-api-access-wc6n7 DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bee40e70-1cd3-4b89-8808-3abf1dcfbee9/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:331 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ebf82283-d275-4640-bc44-5a461e4f9f9d/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:335 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/30c9d23cea06d2e2ee7e26b675a4eadcd7b354fc941609a1dcae96f319a95d40/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9050192a-0ec2-45b9-8b2c-2beb0501b09b/volumes/kubernetes.io~projected/kube-api-access-tv6zc DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ebf82283-d275-4640-bc44-5a461e4f9f9d/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:333 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0308d0c818cb3fa MacAddress:4a:b7:88:70:62:26 Speed:10000 Mtu:8901} {Name:46ee947a9f7dbd2 MacAddress:de:56:88:90:59:f1 Speed:10000 Mtu:8901} {Name:6290d19f6ae8e83 MacAddress:a2:dd:ce:54:8d:b5 Speed:10000 Mtu:8901} {Name:8c6bb37beee1291 MacAddress:66:83:26:01:e6:70 Speed:10000 Mtu:8901} {Name:8fb0cf8cc04cb3d MacAddress:f6:3f:90:8e:50:27 Speed:10000 Mtu:8901} {Name:90cc044bb1c64ad MacAddress:da:d5:a5:f4:33:c2 Speed:10000 Mtu:8901} {Name:92de2ebb594e68d MacAddress:96:4f:19:26:1d:e4 Speed:10000 Mtu:8901} {Name:9f1dde26a34e0e5 MacAddress:1a:03:89:d3:ee:ff Speed:10000 Mtu:8901} {Name:a1513668b128288 MacAddress:62:b5:05:b0:0d:85 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:f7:5e:b1:09 Speed:0 Mtu:9001} {Name:br-int MacAddress:9a:5a:3f:d8:cf:06 Speed:0 Mtu:8901} {Name:d2fdc7bebf927bd MacAddress:ce:41:1c:70:0e:ea Speed:10000 Mtu:8901} {Name:e666b74d740ff3b MacAddress:32:47:f8:dc:1f:fd Speed:10000 Mtu:8901} {Name:eed08506c695a4b MacAddress:22:07:f3:7e:71:47 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:f7:5e:b1:09 Speed:0 Mtu:9001} {Name:f6a2381d6d2ec12 MacAddress:56:01:2e:ec:49:02 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:4a:d6:87:af:50:c5 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ba:9c:27:32:2a:d2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 20:48:25.595676 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.595676 5615 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 20:48:25.596028 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.595721 5615 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 20:48:25.596117 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.596009 5615 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 20:48:25.596211 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.596039 5615 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-192.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 20:48:25.596251 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.596213 5615 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 20:48:25.596251 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.596221 5615 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 20:48:25.596251 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.596232 5615 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:48:25.596251 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.596251 5615 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:48:25.596359 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.596351 5615 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 20:48:25.596530 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.596520 5615 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 20:48:25.596628 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.596619 5615 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 20:48:25.596659 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.596655 5615 kubelet.go:493] "Attempting to sync node with API server" Nov 03 20:48:25.596685 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.596669 5615 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 20:48:25.596685 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.596679 5615 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 20:48:25.596741 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.596688 5615 kubelet.go:405] "Adding apiserver pod source" Nov 03 20:48:25.596741 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.596696 5615 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 20:48:25.597358 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.597345 5615 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 20:48:25.597358 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.597358 5615 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:48:25.597877 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.597861 5615 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 20:48:25.600609 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.600592 5615 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 20:48:25.600609 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.600607 5615 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:48:25.600726 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.600716 5615 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 20:48:25.600939 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.600928 5615 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 20:48:25.600968 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.600957 5615 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 20:48:25.601054 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.601046 5615 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 20:48:25.601093 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.601061 5615 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 20:48:25.601093 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.601067 5615 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 20:48:25.601093 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.601091 5615 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 20:48:25.601173 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.601102 5615 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 20:48:25.601173 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.601111 5615 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 20:48:25.601173 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.601122 5615 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 20:48:25.601173 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.601130 5615 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 20:48:25.601173 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.601140 5615 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 20:48:25.601173 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.601150 5615 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 20:48:25.601173 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.601173 5615 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 20:48:25.601356 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.601185 5615 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 20:48:25.601356 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.601246 5615 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 20:48:25.601356 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.601252 5615 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 20:48:25.604772 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.604751 5615 server.go:1263] "Started kubelet" Nov 03 20:48:25.604864 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.604819 5615 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 20:48:25.605227 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.605054 5615 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 20:48:25.605457 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.605366 5615 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 20:48:25.605460 ip-10-0-129-192 systemd[1]: Started Kubernetes Kubelet. Nov 03 20:48:25.606119 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.605708 5615 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 20:48:25.606268 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.606251 5615 server.go:317] "Adding debug handlers to kubelet server" Nov 03 20:48:25.609830 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.609582 5615 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 20:48:25.613566 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.611173 5615 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 20:48:25.613566 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.611238 5615 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 20:48:25.613566 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.612740 5615 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 20:48:25.613566 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:25.612978 5615 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-192.us-west-2.compute.internal\" not found" Nov 03 20:48:25.614711 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.614085 5615 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 20:48:25.614711 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.614457 5615 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 20:42:15 +0000 UTC" deadline="2028-04-25 17:19:08.704254766 +0000 UTC" Nov 03 20:48:25.614711 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.614607 5615 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12932h30m43.089688111s" Nov 03 20:48:25.615683 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.615504 5615 factory.go:55] Registering systemd factory Nov 03 20:48:25.615683 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.615528 5615 factory.go:223] Registration of the systemd container factory successfully Nov 03 20:48:25.616667 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.616649 5615 factory.go:153] Registering CRI-O factory Nov 03 20:48:25.616769 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.616670 5615 factory.go:223] Registration of the crio container factory successfully Nov 03 20:48:25.616769 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.616730 5615 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 20:48:25.616769 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.616751 5615 factory.go:103] Registering Raw factory Nov 03 20:48:25.616769 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.616766 5615 manager.go:1199] Started watching for new ooms in manager Nov 03 20:48:25.617177 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.617137 5615 manager.go:319] Starting recovery of all containers Nov 03 20:48:25.619329 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619291 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eaf3e4d3-7019-42e5-9e1e-2927b7f22304" volumeName="kubernetes.io/projected/eaf3e4d3-7019-42e5-9e1e-2927b7f22304-kube-api-access-nkxvg" seLinuxMountContext="" Nov 03 20:48:25.619409 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619329 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2dc043c-52c0-4da0-8370-ed975ebb987f" volumeName="kubernetes.io/projected/a2dc043c-52c0-4da0-8370-ed975ebb987f-kube-api-access-42v78" seLinuxMountContext="" Nov 03 20:48:25.619409 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619344 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05683773-8c40-4d7c-9cc9-ee9d71b25b23" volumeName="kubernetes.io/projected/05683773-8c40-4d7c-9cc9-ee9d71b25b23-kube-api-access-wc6n7" seLinuxMountContext="" Nov 03 20:48:25.619409 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619360 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05683773-8c40-4d7c-9cc9-ee9d71b25b23" volumeName="kubernetes.io/empty-dir/05683773-8c40-4d7c-9cc9-ee9d71b25b23-tmp-dir" seLinuxMountContext="" Nov 03 20:48:25.619409 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619372 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15a5eea7-15b5-45a6-838b-35914cb59be1" volumeName="kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-thanos-querier-tls" seLinuxMountContext="" Nov 03 20:48:25.619409 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619384 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="195d22be-670c-4866-a303-1333085f8a67" volumeName="kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-console-config" seLinuxMountContext="" Nov 03 20:48:25.619409 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619396 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0327c758-d1f5-481d-9cf2-96e4b5f27872" volumeName="kubernetes.io/secret/0327c758-d1f5-481d-9cf2-96e4b5f27872-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 20:48:25.619409 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619409 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05683773-8c40-4d7c-9cc9-ee9d71b25b23" volumeName="kubernetes.io/configmap/05683773-8c40-4d7c-9cc9-ee9d71b25b23-config-volume" seLinuxMountContext="" Nov 03 20:48:25.619753 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619424 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15a5eea7-15b5-45a6-838b-35914cb59be1" volumeName="kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 03 20:48:25.619753 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619436 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9050192a-0ec2-45b9-8b2c-2beb0501b09b" volumeName="kubernetes.io/empty-dir/9050192a-0ec2-45b9-8b2c-2beb0501b09b-tmp-dir" seLinuxMountContext="" Nov 03 20:48:25.619753 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619447 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a882e61f-2b83-4ff4-8370-cba401a6d9c8" volumeName="kubernetes.io/configmap/a882e61f-2b83-4ff4-8370-cba401a6d9c8-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 20:48:25.619753 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619459 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c14bf8f0-1335-4ee0-b49d-5d1f2836de1c" volumeName="kubernetes.io/empty-dir/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-volume-directive-shadow" seLinuxMountContext="" Nov 03 20:48:25.619753 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619472 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c14bf8f0-1335-4ee0-b49d-5d1f2836de1c" volumeName="kubernetes.io/projected/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-kube-api-access-fbvc2" seLinuxMountContext="" Nov 03 20:48:25.619753 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619483 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fac5da49-6649-4779-9529-089804436313" volumeName="kubernetes.io/projected/fac5da49-6649-4779-9529-089804436313-kube-api-access-k85jh" seLinuxMountContext="" Nov 03 20:48:25.619753 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619523 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0327c758-d1f5-481d-9cf2-96e4b5f27872" volumeName="kubernetes.io/configmap/0327c758-d1f5-481d-9cf2-96e4b5f27872-telemeter-trusted-ca-bundle" seLinuxMountContext="" Nov 03 20:48:25.619753 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619536 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3e654a3-7d76-4793-9627-6334ba6db6d9" volumeName="kubernetes.io/projected/a3e654a3-7d76-4793-9627-6334ba6db6d9-kube-api-access-prrnl" seLinuxMountContext="" Nov 03 20:48:25.619753 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619549 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c77a4d5b-a0d3-468b-aa66-7d607794aba1" volumeName="kubernetes.io/projected/c77a4d5b-a0d3-468b-aa66-7d607794aba1-kube-api-access-9pgk2" seLinuxMountContext="" Nov 03 20:48:25.619753 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619561 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e64dc44d-5470-4abb-8155-fdf7f24c253b" volumeName="kubernetes.io/configmap/e64dc44d-5470-4abb-8155-fdf7f24c253b-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 20:48:25.619753 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619573 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0327c758-d1f5-481d-9cf2-96e4b5f27872" volumeName="kubernetes.io/projected/0327c758-d1f5-481d-9cf2-96e4b5f27872-kube-api-access-p7rds" seLinuxMountContext="" Nov 03 20:48:25.619753 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619596 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a695777-f42a-4cb6-8a31-c07c8daf321a" volumeName="kubernetes.io/empty-dir/3a695777-f42a-4cb6-8a31-c07c8daf321a-node-exporter-textfile" seLinuxMountContext="" Nov 03 20:48:25.619753 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619607 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81" volumeName="kubernetes.io/projected/955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81-kube-api-access-j5mgg" seLinuxMountContext="" Nov 03 20:48:25.619753 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619619 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a882e61f-2b83-4ff4-8370-cba401a6d9c8" volumeName="kubernetes.io/projected/a882e61f-2b83-4ff4-8370-cba401a6d9c8-kube-api-access-6ttzr" seLinuxMountContext="" Nov 03 20:48:25.619753 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619630 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c14bf8f0-1335-4ee0-b49d-5d1f2836de1c" volumeName="kubernetes.io/configmap/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 03 20:48:25.619753 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619641 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eaf3e4d3-7019-42e5-9e1e-2927b7f22304" volumeName="kubernetes.io/configmap/eaf3e4d3-7019-42e5-9e1e-2927b7f22304-iptables-alerter-script" seLinuxMountContext="" Nov 03 20:48:25.619753 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619654 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="195d22be-670c-4866-a303-1333085f8a67" volumeName="kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-oauth-serving-cert" seLinuxMountContext="" Nov 03 20:48:25.619753 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619666 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="026b5dc8-bcff-4afc-a784-baf95c1cec27" volumeName="kubernetes.io/empty-dir/026b5dc8-bcff-4afc-a784-baf95c1cec27-tmp" seLinuxMountContext="" Nov 03 20:48:25.619753 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619690 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0327c758-d1f5-481d-9cf2-96e4b5f27872" volumeName="kubernetes.io/secret/0327c758-d1f5-481d-9cf2-96e4b5f27872-telemeter-client-tls" seLinuxMountContext="" Nov 03 20:48:25.619753 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619704 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a882e61f-2b83-4ff4-8370-cba401a6d9c8" volumeName="kubernetes.io/configmap/a882e61f-2b83-4ff4-8370-cba401a6d9c8-cni-binary-copy" seLinuxMountContext="" Nov 03 20:48:25.619753 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619720 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfca9b06-e0d2-4d0f-a75c-7827485cb42d" volumeName="kubernetes.io/configmap/bfca9b06-e0d2-4d0f-a75c-7827485cb42d-serviceca" seLinuxMountContext="" Nov 03 20:48:25.619753 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619742 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c" volumeName="kubernetes.io/projected/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-kube-api-access-dh5qw" seLinuxMountContext="" Nov 03 20:48:25.619753 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619763 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e64dc44d-5470-4abb-8155-fdf7f24c253b" volumeName="kubernetes.io/secret/e64dc44d-5470-4abb-8155-fdf7f24c253b-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619786 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebf82283-d275-4640-bc44-5a461e4f9f9d" volumeName="kubernetes.io/configmap/ebf82283-d275-4640-bc44-5a461e4f9f9d-metrics-server-audit-profiles" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619800 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a695777-f42a-4cb6-8a31-c07c8daf321a" volumeName="kubernetes.io/secret/3a695777-f42a-4cb6-8a31-c07c8daf321a-node-exporter-tls" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619820 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a882e61f-2b83-4ff4-8370-cba401a6d9c8" volumeName="kubernetes.io/configmap/a882e61f-2b83-4ff4-8370-cba401a6d9c8-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619833 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c77a4d5b-a0d3-468b-aa66-7d607794aba1" volumeName="kubernetes.io/secret/c77a4d5b-a0d3-468b-aa66-7d607794aba1-openshift-state-metrics-tls" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619846 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebf82283-d275-4640-bc44-5a461e4f9f9d" volumeName="kubernetes.io/empty-dir/ebf82283-d275-4640-bc44-5a461e4f9f9d-audit-log" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619868 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0327c758-d1f5-481d-9cf2-96e4b5f27872" volumeName="kubernetes.io/configmap/0327c758-d1f5-481d-9cf2-96e4b5f27872-serving-certs-ca-bundle" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619879 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0327c758-d1f5-481d-9cf2-96e4b5f27872" volumeName="kubernetes.io/secret/0327c758-d1f5-481d-9cf2-96e4b5f27872-secret-telemeter-client" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619891 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="195d22be-670c-4866-a303-1333085f8a67" volumeName="kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-trusted-ca-bundle" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619910 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a695777-f42a-4cb6-8a31-c07c8daf321a" volumeName="kubernetes.io/configmap/3a695777-f42a-4cb6-8a31-c07c8daf321a-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619932 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="712fcecb-4757-4dd0-8f96-084b1e6ee903" volumeName="kubernetes.io/secret/712fcecb-4757-4dd0-8f96-084b1e6ee903-original-pull-secret" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619945 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2dc043c-52c0-4da0-8370-ed975ebb987f" volumeName="kubernetes.io/configmap/a2dc043c-52c0-4da0-8370-ed975ebb987f-env-overrides" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619959 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05683773-8c40-4d7c-9cc9-ee9d71b25b23" volumeName="kubernetes.io/secret/05683773-8c40-4d7c-9cc9-ee9d71b25b23-metrics-tls" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619972 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="195d22be-670c-4866-a303-1333085f8a67" volumeName="kubernetes.io/projected/195d22be-670c-4866-a303-1333085f8a67-kube-api-access-vw4cq" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619984 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebf82283-d275-4640-bc44-5a461e4f9f9d" volumeName="kubernetes.io/projected/ebf82283-d275-4640-bc44-5a461e4f9f9d-kube-api-access-pbsdk" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.619997 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebf82283-d275-4640-bc44-5a461e4f9f9d" volumeName="kubernetes.io/secret/ebf82283-d275-4640-bc44-5a461e4f9f9d-client-ca-bundle" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620010 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afd8ecac-e0b7-4453-a4b5-c17ba6504cbb" volumeName="kubernetes.io/projected/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-kube-api-access-cbctt" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620023 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="026b5dc8-bcff-4afc-a784-baf95c1cec27" volumeName="kubernetes.io/empty-dir/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-tuned" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620041 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15a5eea7-15b5-45a6-838b-35914cb59be1" volumeName="kubernetes.io/configmap/15a5eea7-15b5-45a6-838b-35914cb59be1-metrics-client-ca" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620053 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="195d22be-670c-4866-a303-1333085f8a67" volumeName="kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-serving-cert" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620066 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9050192a-0ec2-45b9-8b2c-2beb0501b09b" volumeName="kubernetes.io/projected/9050192a-0ec2-45b9-8b2c-2beb0501b09b-kube-api-access-tv6zc" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620096 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfca9b06-e0d2-4d0f-a75c-7827485cb42d" volumeName="kubernetes.io/projected/bfca9b06-e0d2-4d0f-a75c-7827485cb42d-kube-api-access-tf5tz" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620108 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c77a4d5b-a0d3-468b-aa66-7d607794aba1" volumeName="kubernetes.io/configmap/c77a4d5b-a0d3-468b-aa66-7d607794aba1-metrics-client-ca" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620121 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cba96b6a-ccc8-4c1c-b371-1bb5ece29afc" volumeName="kubernetes.io/configmap/cba96b6a-ccc8-4c1c-b371-1bb5ece29afc-konnectivity-ca" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620136 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15a5eea7-15b5-45a6-838b-35914cb59be1" volumeName="kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620149 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afd8ecac-e0b7-4453-a4b5-c17ba6504cbb" volumeName="kubernetes.io/configmap/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-multus-daemon-config" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620177 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afd8ecac-e0b7-4453-a4b5-c17ba6504cbb" volumeName="kubernetes.io/configmap/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-cni-binary-copy" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620189 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bee40e70-1cd3-4b89-8808-3abf1dcfbee9" volumeName="kubernetes.io/secret/bee40e70-1cd3-4b89-8808-3abf1dcfbee9-monitoring-plugin-cert" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620203 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c" volumeName="kubernetes.io/secret/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-metrics-certs" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620219 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e64dc44d-5470-4abb-8155-fdf7f24c253b" volumeName="kubernetes.io/empty-dir/e64dc44d-5470-4abb-8155-fdf7f24c253b-data-volume" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620234 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e64dc44d-5470-4abb-8155-fdf7f24c253b" volumeName="kubernetes.io/projected/e64dc44d-5470-4abb-8155-fdf7f24c253b-kube-api-access-zzx44" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620263 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebf82283-d275-4640-bc44-5a461e4f9f9d" volumeName="kubernetes.io/secret/ebf82283-d275-4640-bc44-5a461e4f9f9d-secret-metrics-server-tls" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620276 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0327c758-d1f5-481d-9cf2-96e4b5f27872" volumeName="kubernetes.io/configmap/0327c758-d1f5-481d-9cf2-96e4b5f27872-metrics-client-ca" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620297 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15a5eea7-15b5-45a6-838b-35914cb59be1" volumeName="kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620313 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a695777-f42a-4cb6-8a31-c07c8daf321a" volumeName="kubernetes.io/secret/3a695777-f42a-4cb6-8a31-c07c8daf321a-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620326 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bee40e70-1cd3-4b89-8808-3abf1dcfbee9" volumeName="kubernetes.io/projected/bee40e70-1cd3-4b89-8808-3abf1dcfbee9-kube-api-access-rbhtz" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620338 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebf82283-d275-4640-bc44-5a461e4f9f9d" volumeName="kubernetes.io/configmap/ebf82283-d275-4640-bc44-5a461e4f9f9d-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620350 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fac5da49-6649-4779-9529-089804436313" volumeName="kubernetes.io/secret/fac5da49-6649-4779-9529-089804436313-cert" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620360 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0327c758-d1f5-481d-9cf2-96e4b5f27872" volumeName="kubernetes.io/secret/0327c758-d1f5-481d-9cf2-96e4b5f27872-federate-client-tls" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620371 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="195d22be-670c-4866-a303-1333085f8a67" volumeName="kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-oauth-config" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620384 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a695777-f42a-4cb6-8a31-c07c8daf321a" volumeName="kubernetes.io/configmap/3a695777-f42a-4cb6-8a31-c07c8daf321a-metrics-client-ca" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620397 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="026b5dc8-bcff-4afc-a784-baf95c1cec27" volumeName="kubernetes.io/projected/026b5dc8-bcff-4afc-a784-baf95c1cec27-kube-api-access-p6lpt" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620409 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15a5eea7-15b5-45a6-838b-35914cb59be1" volumeName="kubernetes.io/projected/15a5eea7-15b5-45a6-838b-35914cb59be1-kube-api-access-x6x9d" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620419 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15a5eea7-15b5-45a6-838b-35914cb59be1" volumeName="kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-grpc-tls" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620430 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15a5eea7-15b5-45a6-838b-35914cb59be1" volumeName="kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620445 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2dc043c-52c0-4da0-8370-ed975ebb987f" volumeName="kubernetes.io/secret/a2dc043c-52c0-4da0-8370-ed975ebb987f-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620456 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15a5eea7-15b5-45a6-838b-35914cb59be1" volumeName="kubernetes.io/empty-dir/15a5eea7-15b5-45a6-838b-35914cb59be1-tmp" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620467 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="195d22be-670c-4866-a303-1333085f8a67" volumeName="kubernetes.io/empty-dir/195d22be-670c-4866-a303-1333085f8a67-tmp" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620480 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a695777-f42a-4cb6-8a31-c07c8daf321a" volumeName="kubernetes.io/projected/3a695777-f42a-4cb6-8a31-c07c8daf321a-kube-api-access-lhkvl" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620495 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c14bf8f0-1335-4ee0-b49d-5d1f2836de1c" volumeName="kubernetes.io/configmap/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-metrics-client-ca" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620509 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c14bf8f0-1335-4ee0-b49d-5d1f2836de1c" volumeName="kubernetes.io/secret/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620521 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c14bf8f0-1335-4ee0-b49d-5d1f2836de1c" volumeName="kubernetes.io/secret/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-kube-state-metrics-tls" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620532 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c77a4d5b-a0d3-468b-aa66-7d607794aba1" volumeName="kubernetes.io/secret/c77a4d5b-a0d3-468b-aa66-7d607794aba1-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620543 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebf82283-d275-4640-bc44-5a461e4f9f9d" volumeName="kubernetes.io/secret/ebf82283-d275-4640-bc44-5a461e4f9f9d-secret-metrics-server-client-certs" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620556 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="195d22be-670c-4866-a303-1333085f8a67" volumeName="kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-service-ca" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620568 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2dc043c-52c0-4da0-8370-ed975ebb987f" volumeName="kubernetes.io/configmap/a2dc043c-52c0-4da0-8370-ed975ebb987f-ovnkube-config" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620581 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2dc043c-52c0-4da0-8370-ed975ebb987f" volumeName="kubernetes.io/configmap/a2dc043c-52c0-4da0-8370-ed975ebb987f-ovnkube-script-lib" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620594 5615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cba96b6a-ccc8-4c1c-b371-1bb5ece29afc" volumeName="kubernetes.io/secret/cba96b6a-ccc8-4c1c-b371-1bb5ece29afc-agent-certs" seLinuxMountContext="" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620606 5615 reconstruct.go:97] "Volume reconstruction finished" Nov 03 20:48:25.620803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.620615 5615 reconciler.go:29] "Reconciler: start to sync state" Nov 03 20:48:25.641921 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.641888 5615 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 20:48:25.643576 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.643541 5615 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 20:48:25.643576 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.643575 5615 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 20:48:25.643761 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.643596 5615 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 20:48:25.643761 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:25.643637 5615 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 20:48:25.680647 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.680586 5615 generic.go:356] "Generic (PLEG): container finished" podID="a882e61f-2b83-4ff4-8370-cba401a6d9c8" containerID="9365311a852bade283bae3aa2130a09998d0e2075ec725e0691a0c872992072d" exitCode=0 Nov 03 20:48:25.680647 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.680638 5615 generic.go:356] "Generic (PLEG): container finished" podID="a882e61f-2b83-4ff4-8370-cba401a6d9c8" containerID="d54c5c98b47d88b4fbfe5ad565ae0c278bd92030464c7a77fa8c1c1d8e1ad515" exitCode=0 Nov 03 20:48:25.680647 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.680651 5615 generic.go:356] "Generic (PLEG): container finished" podID="a882e61f-2b83-4ff4-8370-cba401a6d9c8" containerID="29dcb550d7251ec444ded1fe2ef043332cb33107a2df27b4f438d10d583800a5" exitCode=0 Nov 03 20:48:25.680897 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.680661 5615 generic.go:356] "Generic (PLEG): container finished" podID="a882e61f-2b83-4ff4-8370-cba401a6d9c8" containerID="567443ba70810871551abbc60ed13abaae9134401d1d69cb78e333afec118431" exitCode=0 Nov 03 20:48:25.680897 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.680671 5615 generic.go:356] "Generic (PLEG): container finished" podID="a882e61f-2b83-4ff4-8370-cba401a6d9c8" containerID="7c3a4eb1e5feb2f6c8f7a0139cedc0cae97ecd31d38d35e6fd76f1e0bd052e31" exitCode=0 Nov 03 20:48:25.680897 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.680680 5615 generic.go:356] "Generic (PLEG): container finished" podID="a882e61f-2b83-4ff4-8370-cba401a6d9c8" containerID="85af365816543163ab496b1e4f614b0508143235c68f5486fd9ed1dec0c68d12" exitCode=0 Nov 03 20:48:25.693519 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.693479 5615 generic.go:356] "Generic (PLEG): container finished" podID="3a695777-f42a-4cb6-8a31-c07c8daf321a" containerID="9d84ce16ea8218efd6a8292e83042ca841179f25f26f79f59eb17634644226b2" exitCode=0 Nov 03 20:48:25.701000 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.700972 5615 generic.go:356] "Generic (PLEG): container finished" podID="55575692b9132c5959a94942a022491a" containerID="a661acc6ec95514f9d77dcd49a98d8a6749e32fbac985a9f2bf140465bfbcfb4" exitCode=0 Nov 03 20:48:25.713102 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:25.713046 5615 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-192.us-west-2.compute.internal\" not found" Nov 03 20:48:25.725477 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.725414 5615 manager.go:324] Recovery completed Nov 03 20:48:25.744156 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:25.744114 5615 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 20:48:25.746703 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.746687 5615 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:48:25.748837 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.748806 5615 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:48:25.748837 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.748842 5615 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:48:25.748980 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.748853 5615 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:48:25.752127 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.752110 5615 cpu_manager.go:226] "Starting" policy="none" Nov 03 20:48:25.752127 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.752123 5615 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 20:48:25.752239 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.752143 5615 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 20:48:25.752239 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.752124 5615 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:48:25.752320 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.752266 5615 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 20:48:25.752320 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.752277 5615 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 20:48:25.752320 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.752300 5615 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 20:48:25.752320 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.752305 5615 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 20:48:25.752320 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.752310 5615 policy_none.go:50] "Start" Nov 03 20:48:25.752320 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.752319 5615 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 20:48:25.752486 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.752328 5615 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 20:48:25.752486 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.752453 5615 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 20:48:25.752486 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.752460 5615 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 20:48:25.752486 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.752466 5615 policy_none.go:44] "Start" Nov 03 20:48:25.753252 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.753236 5615 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:48:25.753312 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.753267 5615 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:48:25.753312 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.753277 5615 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:48:25.757267 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.757238 5615 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 20:48:25.761790 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.761775 5615 manager.go:341] "Starting Device Plugin manager" Nov 03 20:48:25.761889 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:25.761805 5615 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 20:48:25.761889 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.761819 5615 server.go:87] "Starting device plugin registration server" Nov 03 20:48:25.762146 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.762133 5615 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 20:48:25.762224 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.762147 5615 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 20:48:25.762278 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.762225 5615 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 20:48:25.762359 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.762325 5615 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 20:48:25.762405 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.762375 5615 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 20:48:25.762405 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.762383 5615 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 20:48:25.762512 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.762434 5615 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:48:25.762358518Z","UUID":"af1b7631-3efb-4d1a-bcde-34f85c6be25b","Handler":null,"Name":"","Endpoint":""} Nov 03 20:48:25.763841 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.763818 5615 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 20:48:25.763921 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.763857 5615 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 20:48:25.764287 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:25.764262 5615 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 20:48:25.764364 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:25.764321 5615 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-192.us-west-2.compute.internal\" not found" Nov 03 20:48:25.797875 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.797837 5615 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:48:25.797875 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.797851 5615 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:48:25.798038 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.797923 5615 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 20:48:25.800684 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.800657 5615 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:48:25.862461 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.862431 5615 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:48:25.863629 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.863611 5615 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:48:25.863699 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.863643 5615 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:48:25.863699 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.863654 5615 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-192.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:48:25.863699 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.863699 5615 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:48:25.891821 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.891796 5615 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:48:25.944774 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.944727 5615 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal"] Nov 03 20:48:25.945480 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.945463 5615 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:48:25.945620 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.945579 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal" event={"ID":"448c85734c0da9850a3328446f82dce9","Type":"ContainerStarted","Data":"50ea737e370d91b3dd5eba1bd2d00c85aeb83f4a79c43b4c678e63b22baf2c58"} Nov 03 20:48:25.945654 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.945636 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal" event={"ID":"448c85734c0da9850a3328446f82dce9","Type":"ContainerStarted","Data":"740a6d0a1a116d13d11958b6f022a8a647d699b659a1b9b83ee9a8acae3b8e22"} Nov 03 20:48:25.945697 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.945681 5615 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:48:25.945749 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.945683 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal" event={"ID":"55575692b9132c5959a94942a022491a","Type":"ContainerStarted","Data":"f332c07dc1966a69ec1591df1fc28ff79d4e32a9a21d276aa50baea5ae4eebfc"} Nov 03 20:48:25.945749 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.945719 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal" event={"ID":"55575692b9132c5959a94942a022491a","Type":"ContainerDied","Data":"a661acc6ec95514f9d77dcd49a98d8a6749e32fbac985a9f2bf140465bfbcfb4"} Nov 03 20:48:25.945749 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.945731 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal" event={"ID":"55575692b9132c5959a94942a022491a","Type":"ContainerStarted","Data":"f60e0f8c9bd1538eeca368800ccd9da47cda90699ebc3e92d80a6a7b3ec38513"} Nov 03 20:48:25.968585 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.968562 5615 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:48:25.968691 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:25.968630 5615 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:48:26.003605 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.003579 5615 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:48:26.003742 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:26.003626 5615 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:48:26.006392 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.006373 5615 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:48:26.006467 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:26.006412 5615 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:48:26.099044 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.099011 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/55575692b9132c5959a94942a022491a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal\" (UID: \"55575692b9132c5959a94942a022491a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:48:26.099044 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.099044 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55575692b9132c5959a94942a022491a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal\" (UID: \"55575692b9132c5959a94942a022491a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:48:26.099269 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.099061 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/448c85734c0da9850a3328446f82dce9-config\") pod \"kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal\" (UID: \"448c85734c0da9850a3328446f82dce9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:48:26.199234 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.199194 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/55575692b9132c5959a94942a022491a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal\" (UID: \"55575692b9132c5959a94942a022491a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:48:26.199234 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.199246 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55575692b9132c5959a94942a022491a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal\" (UID: \"55575692b9132c5959a94942a022491a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:48:26.199446 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.199276 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/55575692b9132c5959a94942a022491a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal\" (UID: \"55575692b9132c5959a94942a022491a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:48:26.199446 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.199306 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55575692b9132c5959a94942a022491a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal\" (UID: \"55575692b9132c5959a94942a022491a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:48:26.199446 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.199338 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/448c85734c0da9850a3328446f82dce9-config\") pod \"kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal\" (UID: \"448c85734c0da9850a3328446f82dce9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:48:26.199446 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.199381 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/448c85734c0da9850a3328446f82dce9-config\") pod \"kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal\" (UID: \"448c85734c0da9850a3328446f82dce9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:48:26.596952 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.596918 5615 apiserver.go:52] "Watching apiserver" Nov 03 20:48:26.704708 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.704679 5615 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:48:26.704839 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.704778 5615 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:48:26.742116 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.742066 5615 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 20:48:26.742727 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.742701 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-lm85p","openshift-dns/node-resolver-vq2rj","openshift-console/console-7f847c9697-nsrdm","openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6","openshift-multus/multus-thxwg","openshift-ovn-kubernetes/ovnkube-node-mfbd2","kube-system/kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal","openshift-monitoring/monitoring-plugin-77f4f76594-n8k2s","openshift-monitoring/thanos-querier-5d578896-ftx7j","kube-system/global-pull-secret-syncer-mqmds","openshift-multus/network-metrics-daemon-tqhnr","openshift-ingress-canary/ingress-canary-7vv52","openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96","openshift-image-registry/node-ca-lmq6t","openshift-monitoring/metrics-server-b9b549fcc-4cxct","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc","openshift-monitoring/node-exporter-ffhc8","openshift-network-operator/iptables-alerter-9bt29","openshift-dns/dns-default-7cvm2","kube-system/konnectivity-agent-ksg72","openshift-insights/insights-runtime-extractor-tms4q","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-qwqjp","openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f","openshift-multus/multus-additional-cni-plugins-5p62l"] Nov 03 20:48:26.762803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.762779 5615 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:48:26.762931 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:26.762837 5615 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:48:26.763228 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.763215 5615 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:48:26.763301 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:26.763260 5615 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.us-west-2.compute.internal" Nov 03 20:48:26.794157 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.794121 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tlf9q\"" Nov 03 20:48:26.794157 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.794146 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vwxfp\"" Nov 03 20:48:26.794409 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.794286 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-r6str\"" Nov 03 20:48:26.794409 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.794287 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-snm99\"" Nov 03 20:48:26.794409 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.794306 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 20:48:26.794409 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.794371 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 20:48:26.794409 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.794371 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 20:48:26.794702 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.794423 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 20:48:26.794702 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.794423 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:48:26.794702 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.794495 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 20:48:26.794702 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.794526 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:48:26.794702 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.794527 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 20:48:26.794702 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.794544 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 20:48:26.794702 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.794578 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 20:48:26.794702 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.794654 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 20:48:26.794702 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.794623 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 20:48:26.795153 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.794719 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 20:48:26.795153 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.794766 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 20:48:26.795153 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.794784 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 20:48:26.795153 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.794828 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xldt8\"" Nov 03 20:48:26.795153 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.794834 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 20:48:26.795153 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.794901 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ldspv\"" Nov 03 20:48:26.795153 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.794915 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 20:48:26.795153 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.794968 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 20:48:26.795153 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.795032 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 20:48:26.795153 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.795044 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gf6gt\"" Nov 03 20:48:26.795153 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.795141 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 20:48:26.802743 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.802719 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 20:48:26.803045 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.803022 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 20:48:26.803177 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.803037 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-68sfv\"" Nov 03 20:48:26.803177 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.803099 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 20:48:26.803412 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.803332 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4vnfl\"" Nov 03 20:48:26.803412 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.803359 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 20:48:26.803412 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.803372 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 20:48:26.803412 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.803371 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-lhhqb\"" Nov 03 20:48:26.803584 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.803427 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 20:48:26.803584 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.803555 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 20:48:26.803683 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.803585 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 20:48:26.803787 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.803703 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 20:48:26.803787 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.803764 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 20:48:26.803860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.803789 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 20:48:26.803912 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.803887 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 20:48:26.803912 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.803893 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 20:48:26.804002 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.803924 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 20:48:26.804002 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.803950 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 20:48:26.804002 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.803988 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 20:48:26.804143 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.804104 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 20:48:26.804249 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.804234 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 20:48:26.804323 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.804260 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-d7jn9\"" Nov 03 20:48:26.804323 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.804278 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 20:48:26.804323 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.804288 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 20:48:26.804446 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.804361 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 20:48:26.804446 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.804400 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 20:48:26.804446 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.804407 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 20:48:26.804446 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.804409 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-868xx\"" Nov 03 20:48:26.804446 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.804415 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-r6h6b\"" Nov 03 20:48:26.804446 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.804428 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7vv6z\"" Nov 03 20:48:26.804741 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.804490 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 20:48:26.804861 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.804847 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2g8j2\"" Nov 03 20:48:26.810264 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.810249 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 20:48:26.810264 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.810257 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 20:48:26.810368 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.810251 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l9gsz\"" Nov 03 20:48:26.810368 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.810280 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-78pt4\"" Nov 03 20:48:26.810368 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.810287 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 20:48:26.810368 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.810287 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 20:48:26.810368 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.810289 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mxhpl\"" Nov 03 20:48:26.810368 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.810363 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 20:48:26.810559 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.810533 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 20:48:26.816441 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.816416 5615 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 20:48:26.840508 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.840480 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 20:48:26.841148 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.841132 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 20:48:26.842544 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.842385 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 03 20:48:26.842544 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.842388 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 20:48:26.842544 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.842446 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 20:48:26.842544 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.842446 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-cbbfr\"" Nov 03 20:48:26.842544 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.842501 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 20:48:26.842544 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.842505 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 03 20:48:26.842544 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.842392 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 20:48:26.843110 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.842590 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 20:48:26.843110 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.842858 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 20:48:26.843110 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.842909 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 03 20:48:26.843110 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.842941 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 03 20:48:26.843110 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.843022 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8bglh\"" Nov 03 20:48:26.843110 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.843042 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 20:48:26.843283 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.843139 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-bst6w\"" Nov 03 20:48:26.843704 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.843682 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 03 20:48:26.894548 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.894431 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 20:48:26.894548 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.894474 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 03 20:48:26.894704 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.894621 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 20:48:26.894704 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.894642 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-86be21572r5l4\"" Nov 03 20:48:26.894704 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.894662 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-k7jkq\"" Nov 03 20:48:26.894794 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.894719 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-kth16345cnsc\"" Nov 03 20:48:26.903893 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.903869 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbhtz\" (UniqueName: \"kubernetes.io/projected/bee40e70-1cd3-4b89-8808-3abf1dcfbee9-kube-api-access-rbhtz\") pod \"monitoring-plugin-77f4f76594-n8k2s\" (UID: \"bee40e70-1cd3-4b89-8808-3abf1dcfbee9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-n8k2s" Nov 03 20:48:26.903946 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.903901 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a695777-f42a-4cb6-8a31-c07c8daf321a-sys\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:26.903946 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.903919 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9050192a-0ec2-45b9-8b2c-2beb0501b09b-hosts-file\") pod \"node-resolver-vq2rj\" (UID: \"9050192a-0ec2-45b9-8b2c-2beb0501b09b\") " pod="openshift-dns/node-resolver-vq2rj" Nov 03 20:48:26.904020 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.903948 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e64dc44d-5470-4abb-8155-fdf7f24c253b-data-volume\") pod \"insights-runtime-extractor-tms4q\" (UID: \"e64dc44d-5470-4abb-8155-fdf7f24c253b\") " pod="openshift-insights/insights-runtime-extractor-tms4q" Nov 03 20:48:26.904020 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.903974 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:26.904150 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904015 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ebf82283-d275-4640-bc44-5a461e4f9f9d-audit-log\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:26.904150 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904068 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0327c758-d1f5-481d-9cf2-96e4b5f27872-federate-client-tls\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:26.904238 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904158 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-log-socket\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:26.904238 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904172 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e64dc44d-5470-4abb-8155-fdf7f24c253b-data-volume\") pod \"insights-runtime-extractor-tms4q\" (UID: \"e64dc44d-5470-4abb-8155-fdf7f24c253b\") " pod="openshift-insights/insights-runtime-extractor-tms4q" Nov 03 20:48:26.904238 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904192 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a882e61f-2b83-4ff4-8370-cba401a6d9c8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:48:26.904238 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904223 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ebf82283-d275-4640-bc44-5a461e4f9f9d-secret-metrics-server-tls\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:26.904420 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904263 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-multus-cni-dir\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:26.904420 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904330 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:26.904420 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904361 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0327c758-d1f5-481d-9cf2-96e4b5f27872-federate-client-tls\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:26.904420 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904382 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-z8p96\" (UID: \"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:26.904420 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904385 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ebf82283-d275-4640-bc44-5a461e4f9f9d-audit-log\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:26.904648 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904420 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0327c758-d1f5-481d-9cf2-96e4b5f27872-serving-certs-ca-bundle\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:26.904648 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904427 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ebf82283-d275-4640-bc44-5a461e4f9f9d-secret-metrics-server-tls\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:26.904648 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904497 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-run-openvswitch\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:26.904648 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904520 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a882e61f-2b83-4ff4-8370-cba401a6d9c8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:48:26.904648 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904542 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3a695777-f42a-4cb6-8a31-c07c8daf321a-node-exporter-accelerators-collector-config\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:26.904648 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904566 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0327c758-d1f5-481d-9cf2-96e4b5f27872-serving-certs-ca-bundle\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:26.904648 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904581 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhkvl\" (UniqueName: \"kubernetes.io/projected/3a695777-f42a-4cb6-8a31-c07c8daf321a-kube-api-access-lhkvl\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:26.904648 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904603 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-z8p96\" (UID: \"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:26.904648 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904630 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-kubelet\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:26.904648 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904650 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-slash\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:26.905043 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904693 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ebf82283-d275-4640-bc44-5a461e4f9f9d-metrics-server-audit-profiles\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:26.905043 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904720 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-z8p96\" (UID: \"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:26.905043 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904747 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw4cq\" (UniqueName: \"kubernetes.io/projected/195d22be-670c-4866-a303-1333085f8a67-kube-api-access-vw4cq\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:26.905043 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904796 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-registration-dir\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:48:26.905043 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904809 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-z8p96\" (UID: \"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:26.905043 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904823 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:48:26.905043 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904853 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-run-multus-certs\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:26.905043 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904859 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3a695777-f42a-4cb6-8a31-c07c8daf321a-node-exporter-accelerators-collector-config\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:26.905043 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904901 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a2dc043c-52c0-4da0-8370-ed975ebb987f-env-overrides\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:26.905043 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904931 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k85jh\" (UniqueName: \"kubernetes.io/projected/fac5da49-6649-4779-9529-089804436313-kube-api-access-k85jh\") pod \"ingress-canary-7vv52\" (UID: \"fac5da49-6649-4779-9529-089804436313\") " pod="openshift-ingress-canary/ingress-canary-7vv52" Nov 03 20:48:26.905043 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904931 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ebf82283-d275-4640-bc44-5a461e4f9f9d-metrics-server-audit-profiles\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:26.905043 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904949 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/712fcecb-4757-4dd0-8f96-084b1e6ee903-dbus\") pod \"global-pull-secret-syncer-mqmds\" (UID: \"712fcecb-4757-4dd0-8f96-084b1e6ee903\") " pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:48:26.905043 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904967 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a882e61f-2b83-4ff4-8370-cba401a6d9c8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:48:26.905043 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.904984 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6lpt\" (UniqueName: \"kubernetes.io/projected/026b5dc8-bcff-4afc-a784-baf95c1cec27-kube-api-access-p6lpt\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:26.905043 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905000 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-thanos-querier-tls\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:26.905043 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905036 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-device-dir\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:48:26.905759 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905044 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a2dc043c-52c0-4da0-8370-ed975ebb987f-env-overrides\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:26.905759 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905070 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3a695777-f42a-4cb6-8a31-c07c8daf321a-node-exporter-wtmp\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:26.905759 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905144 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-sysctl-conf\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:26.905759 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905183 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv6zc\" (UniqueName: \"kubernetes.io/projected/9050192a-0ec2-45b9-8b2c-2beb0501b09b-kube-api-access-tv6zc\") pod \"node-resolver-vq2rj\" (UID: \"9050192a-0ec2-45b9-8b2c-2beb0501b09b\") " pod="openshift-dns/node-resolver-vq2rj" Nov 03 20:48:26.905759 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905205 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05683773-8c40-4d7c-9cc9-ee9d71b25b23-metrics-tls\") pod \"dns-default-7cvm2\" (UID: \"05683773-8c40-4d7c-9cc9-ee9d71b25b23\") " pod="openshift-dns/dns-default-7cvm2" Nov 03 20:48:26.905759 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905221 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05683773-8c40-4d7c-9cc9-ee9d71b25b23-tmp-dir\") pod \"dns-default-7cvm2\" (UID: \"05683773-8c40-4d7c-9cc9-ee9d71b25b23\") " pod="openshift-dns/dns-default-7cvm2" Nov 03 20:48:26.905759 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905238 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-metrics-certs\") pod \"network-metrics-daemon-tqhnr\" (UID: \"c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c\") " pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:48:26.905759 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905263 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5mgg\" (UniqueName: \"kubernetes.io/projected/955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81-kube-api-access-j5mgg\") pod \"network-check-target-lm85p\" (UID: \"955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81\") " pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:48:26.905759 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905271 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a882e61f-2b83-4ff4-8370-cba401a6d9c8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:48:26.905759 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905279 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-thanos-querier-tls\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:26.905759 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905287 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bfca9b06-e0d2-4d0f-a75c-7827485cb42d-host\") pod \"node-ca-lmq6t\" (UID: \"bfca9b06-e0d2-4d0f-a75c-7827485cb42d\") " pod="openshift-image-registry/node-ca-lmq6t" Nov 03 20:48:26.905759 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905309 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05683773-8c40-4d7c-9cc9-ee9d71b25b23-tmp-dir\") pod \"dns-default-7cvm2\" (UID: \"05683773-8c40-4d7c-9cc9-ee9d71b25b23\") " pod="openshift-dns/dns-default-7cvm2" Nov 03 20:48:26.905759 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905324 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-kubernetes\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:26.905759 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905351 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-tuned\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:26.905759 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905376 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ebf82283-d275-4640-bc44-5a461e4f9f9d-client-ca-bundle\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:26.905759 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905414 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-service-ca\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:26.905759 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905444 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05683773-8c40-4d7c-9cc9-ee9d71b25b23-metrics-tls\") pod \"dns-default-7cvm2\" (UID: \"05683773-8c40-4d7c-9cc9-ee9d71b25b23\") " pod="openshift-dns/dns-default-7cvm2" Nov 03 20:48:26.905759 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:26.905454 5615 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Nov 03 20:48:26.905759 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905465 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-metrics-certs\") pod \"network-metrics-daemon-tqhnr\" (UID: \"c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c\") " pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:48:26.905759 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905486 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-tuned\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:26.905759 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905511 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-etc-kubernetes\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:26.905759 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:26.905549 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-service-ca podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:27.405531983 +0000 UTC m=+1.873310688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-service-ca") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"service-ca" not registered Nov 03 20:48:26.905759 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905567 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42v78\" (UniqueName: \"kubernetes.io/projected/a2dc043c-52c0-4da0-8370-ed975ebb987f-kube-api-access-42v78\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:26.905759 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905601 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fac5da49-6649-4779-9529-089804436313-cert\") pod \"ingress-canary-7vv52\" (UID: \"fac5da49-6649-4779-9529-089804436313\") " pod="openshift-ingress-canary/ingress-canary-7vv52" Nov 03 20:48:26.905759 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905675 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a695777-f42a-4cb6-8a31-c07c8daf321a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:26.905759 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905691 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ebf82283-d275-4640-bc44-5a461e4f9f9d-client-ca-bundle\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:26.905759 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905707 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-z8p96\" (UID: \"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:26.905759 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905736 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a2dc043c-52c0-4da0-8370-ed975ebb987f-ovnkube-config\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:26.905759 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905760 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/712fcecb-4757-4dd0-8f96-084b1e6ee903-original-pull-secret\") pod \"global-pull-secret-syncer-mqmds\" (UID: \"712fcecb-4757-4dd0-8f96-084b1e6ee903\") " pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905790 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a882e61f-2b83-4ff4-8370-cba401a6d9c8-cnibin\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905809 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fac5da49-6649-4779-9529-089804436313-cert\") pod \"ingress-canary-7vv52\" (UID: \"fac5da49-6649-4779-9529-089804436313\") " pod="openshift-ingress-canary/ingress-canary-7vv52" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905836 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a695777-f42a-4cb6-8a31-c07c8daf321a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905899 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a882e61f-2b83-4ff4-8370-cba401a6d9c8-cni-binary-copy\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905932 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e64dc44d-5470-4abb-8155-fdf7f24c253b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tms4q\" (UID: \"e64dc44d-5470-4abb-8155-fdf7f24c253b\") " pod="openshift-insights/insights-runtime-extractor-tms4q" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905950 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ebf82283-d275-4640-bc44-5a461e4f9f9d-secret-metrics-server-client-certs\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905957 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-z8p96\" (UID: \"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905968 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eaf3e4d3-7019-42e5-9e1e-2927b7f22304-host-slash\") pod \"iptables-alerter-9bt29\" (UID: \"eaf3e4d3-7019-42e5-9e1e-2927b7f22304\") " pod="openshift-network-operator/iptables-alerter-9bt29" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905964 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a2dc043c-52c0-4da0-8370-ed975ebb987f-ovnkube-config\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.905993 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0327c758-d1f5-481d-9cf2-96e4b5f27872-telemeter-client-tls\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906033 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3a695777-f42a-4cb6-8a31-c07c8daf321a-root\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906065 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a882e61f-2b83-4ff4-8370-cba401a6d9c8-cni-binary-copy\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906092 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c77a4d5b-a0d3-468b-aa66-7d607794aba1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-zbcg6\" (UID: \"c77a4d5b-a0d3-468b-aa66-7d607794aba1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906128 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0327c758-d1f5-481d-9cf2-96e4b5f27872-secret-telemeter-client\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906130 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e64dc44d-5470-4abb-8155-fdf7f24c253b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tms4q\" (UID: \"e64dc44d-5470-4abb-8155-fdf7f24c253b\") " pod="openshift-insights/insights-runtime-extractor-tms4q" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906159 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/712fcecb-4757-4dd0-8f96-084b1e6ee903-original-pull-secret\") pod \"global-pull-secret-syncer-mqmds\" (UID: \"712fcecb-4757-4dd0-8f96-084b1e6ee903\") " pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906190 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-cni-netd\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906211 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0327c758-d1f5-481d-9cf2-96e4b5f27872-telemeter-client-tls\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906250 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-run\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906299 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c77a4d5b-a0d3-468b-aa66-7d607794aba1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-zbcg6\" (UID: \"c77a4d5b-a0d3-468b-aa66-7d607794aba1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906301 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-var-lib-kubelet\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906332 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ebf82283-d275-4640-bc44-5a461e4f9f9d-secret-metrics-server-client-certs\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906343 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbvc2\" (UniqueName: \"kubernetes.io/projected/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-kube-api-access-fbvc2\") pod \"kube-state-metrics-5dc8d7d89b-z8p96\" (UID: \"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906350 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0327c758-d1f5-481d-9cf2-96e4b5f27872-secret-telemeter-client\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906371 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-oauth-config\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906392 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prrnl\" (UniqueName: \"kubernetes.io/projected/a3e654a3-7d76-4793-9627-6334ba6db6d9-kube-api-access-prrnl\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906409 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-run-netns\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906427 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkxvg\" (UniqueName: \"kubernetes.io/projected/eaf3e4d3-7019-42e5-9e1e-2927b7f22304-kube-api-access-nkxvg\") pod \"iptables-alerter-9bt29\" (UID: \"eaf3e4d3-7019-42e5-9e1e-2927b7f22304\") " pod="openshift-network-operator/iptables-alerter-9bt29" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:26.906458 5615 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906467 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0327c758-d1f5-481d-9cf2-96e4b5f27872-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906512 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7rds\" (UniqueName: \"kubernetes.io/projected/0327c758-d1f5-481d-9cf2-96e4b5f27872-kube-api-access-p7rds\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:26.906533 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-oauth-config podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:27.40651708 +0000 UTC m=+1.874295812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-oauth-config") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"console-oauth-config" not registered Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906563 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-hostroot\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906588 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-run-ovn\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906620 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cba96b6a-ccc8-4c1c-b371-1bb5ece29afc-agent-certs\") pod \"konnectivity-agent-ksg72\" (UID: \"cba96b6a-ccc8-4c1c-b371-1bb5ece29afc\") " pod="kube-system/konnectivity-agent-ksg72" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906649 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a695777-f42a-4cb6-8a31-c07c8daf321a-metrics-client-ca\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906678 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906679 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0327c758-d1f5-481d-9cf2-96e4b5f27872-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906717 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15a5eea7-15b5-45a6-838b-35914cb59be1-metrics-client-ca\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906812 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-etc-openvswitch\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906850 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ttzr\" (UniqueName: \"kubernetes.io/projected/a882e61f-2b83-4ff4-8370-cba401a6d9c8-kube-api-access-6ttzr\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:48:26.906860 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906891 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cba96b6a-ccc8-4c1c-b371-1bb5ece29afc-agent-certs\") pod \"konnectivity-agent-ksg72\" (UID: \"cba96b6a-ccc8-4c1c-b371-1bb5ece29afc\") " pod="kube-system/konnectivity-agent-ksg72" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906903 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-sysconfig\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906959 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15a5eea7-15b5-45a6-838b-35914cb59be1-metrics-client-ca\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906938 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-system-cni-dir\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906980 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906998 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-cni-binary-copy\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.906962 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a695777-f42a-4cb6-8a31-c07c8daf321a-metrics-client-ca\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907015 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-run-netns\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907036 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-node-log\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907092 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bfca9b06-e0d2-4d0f-a75c-7827485cb42d-serviceca\") pod \"node-ca-lmq6t\" (UID: \"bfca9b06-e0d2-4d0f-a75c-7827485cb42d\") " pod="openshift-image-registry/node-ca-lmq6t" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907122 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-multus-socket-dir-parent\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907164 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-cni-binary-copy\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907176 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a882e61f-2b83-4ff4-8370-cba401a6d9c8-system-cni-dir\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907208 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-sysctl-d\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907227 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-console-config\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:26.907299 5615 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907319 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-multus-daemon-config\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907339 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:26.907355 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-console-config podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:27.407341918 +0000 UTC m=+1.875120630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-console-config") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"console-config" not registered Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907393 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e64dc44d-5470-4abb-8155-fdf7f24c253b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tms4q\" (UID: \"e64dc44d-5470-4abb-8155-fdf7f24c253b\") " pod="openshift-insights/insights-runtime-extractor-tms4q" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907424 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pgk2\" (UniqueName: \"kubernetes.io/projected/c77a4d5b-a0d3-468b-aa66-7d607794aba1-kube-api-access-9pgk2\") pod \"openshift-state-metrics-7bd7489957-zbcg6\" (UID: \"c77a4d5b-a0d3-468b-aa66-7d607794aba1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907473 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-multus-daemon-config\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907450 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/05683773-8c40-4d7c-9cc9-ee9d71b25b23-config-volume\") pod \"dns-default-7cvm2\" (UID: \"05683773-8c40-4d7c-9cc9-ee9d71b25b23\") " pod="openshift-dns/dns-default-7cvm2" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907514 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/195d22be-670c-4866-a303-1333085f8a67-tmp\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907541 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-cni-bin\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907566 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-z8p96\" (UID: \"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907591 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-var-lib-cni-multus\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907624 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e64dc44d-5470-4abb-8155-fdf7f24c253b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tms4q\" (UID: \"e64dc44d-5470-4abb-8155-fdf7f24c253b\") " pod="openshift-insights/insights-runtime-extractor-tms4q" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907640 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/195d22be-670c-4866-a303-1333085f8a67-tmp\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907673 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-multus-conf-dir\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907673 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/05683773-8c40-4d7c-9cc9-ee9d71b25b23-config-volume\") pod \"dns-default-7cvm2\" (UID: \"05683773-8c40-4d7c-9cc9-ee9d71b25b23\") " pod="openshift-dns/dns-default-7cvm2" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907734 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbctt\" (UniqueName: \"kubernetes.io/projected/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-kube-api-access-cbctt\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907767 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bfca9b06-e0d2-4d0f-a75c-7827485cb42d-serviceca\") pod \"node-ca-lmq6t\" (UID: \"bfca9b06-e0d2-4d0f-a75c-7827485cb42d\") " pod="openshift-image-registry/node-ca-lmq6t" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907773 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-z8p96\" (UID: \"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907784 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf5tz\" (UniqueName: \"kubernetes.io/projected/bfca9b06-e0d2-4d0f-a75c-7827485cb42d-kube-api-access-tf5tz\") pod \"node-ca-lmq6t\" (UID: \"bfca9b06-e0d2-4d0f-a75c-7827485cb42d\") " pod="openshift-image-registry/node-ca-lmq6t" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907820 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-sys-fs\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907846 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/712fcecb-4757-4dd0-8f96-084b1e6ee903-kubelet-config\") pod \"global-pull-secret-syncer-mqmds\" (UID: \"712fcecb-4757-4dd0-8f96-084b1e6ee903\") " pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907868 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-sys\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907938 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzx44\" (UniqueName: \"kubernetes.io/projected/e64dc44d-5470-4abb-8155-fdf7f24c253b-kube-api-access-zzx44\") pod \"insights-runtime-extractor-tms4q\" (UID: \"e64dc44d-5470-4abb-8155-fdf7f24c253b\") " pod="openshift-insights/insights-runtime-extractor-tms4q" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907965 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-systemd-units\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.907993 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6x9d\" (UniqueName: \"kubernetes.io/projected/15a5eea7-15b5-45a6-838b-35914cb59be1-kube-api-access-x6x9d\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908019 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-cnibin\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908043 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-run-systemd\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908093 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-host\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908195 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-serving-cert\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908217 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-run-k8s-cni-cncf-io\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908241 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-var-lib-kubelet\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908258 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-modprobe-d\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:26.908285 5615 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908302 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e64dc44d-5470-4abb-8155-fdf7f24c253b-crio-socket\") pod \"insights-runtime-extractor-tms4q\" (UID: \"e64dc44d-5470-4abb-8155-fdf7f24c253b\") " pod="openshift-insights/insights-runtime-extractor-tms4q" Nov 03 20:48:26.908317 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:26.908330 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-serving-cert podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:27.408319006 +0000 UTC m=+1.876097725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-serving-cert") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"console-serving-cert" not registered Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908362 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908389 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eaf3e4d3-7019-42e5-9e1e-2927b7f22304-iptables-alerter-script\") pod \"iptables-alerter-9bt29\" (UID: \"eaf3e4d3-7019-42e5-9e1e-2927b7f22304\") " pod="openshift-network-operator/iptables-alerter-9bt29" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908504 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908521 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-oauth-serving-cert\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908566 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-var-lib-cni-bin\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908595 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eaf3e4d3-7019-42e5-9e1e-2927b7f22304-iptables-alerter-script\") pod \"iptables-alerter-9bt29\" (UID: \"eaf3e4d3-7019-42e5-9e1e-2927b7f22304\") " pod="openshift-network-operator/iptables-alerter-9bt29" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908595 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2dc043c-52c0-4da0-8370-ed975ebb987f-ovn-node-metrics-cert\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:26.908626 5615 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908647 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c77a4d5b-a0d3-468b-aa66-7d607794aba1-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-zbcg6\" (UID: \"c77a4d5b-a0d3-468b-aa66-7d607794aba1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908677 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908725 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh5qw\" (UniqueName: \"kubernetes.io/projected/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-kube-api-access-dh5qw\") pod \"network-metrics-daemon-tqhnr\" (UID: \"c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c\") " pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908752 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3a695777-f42a-4cb6-8a31-c07c8daf321a-node-exporter-tls\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908782 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-systemd\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908804 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2dc043c-52c0-4da0-8370-ed975ebb987f-ovn-node-metrics-cert\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:26.908815 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-oauth-serving-cert podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:27.408798238 +0000 UTC m=+1.876576945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-oauth-serving-cert") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"oauth-serving-cert" not registered Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908832 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9050192a-0ec2-45b9-8b2c-2beb0501b09b-tmp-dir\") pod \"node-resolver-vq2rj\" (UID: \"9050192a-0ec2-45b9-8b2c-2beb0501b09b\") " pod="openshift-dns/node-resolver-vq2rj" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908851 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-grpc-tls\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908874 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c77a4d5b-a0d3-468b-aa66-7d607794aba1-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-zbcg6\" (UID: \"c77a4d5b-a0d3-468b-aa66-7d607794aba1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908889 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-run-ovn-kubernetes\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908877 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9050192a-0ec2-45b9-8b2c-2beb0501b09b-tmp-dir\") pod \"node-resolver-vq2rj\" (UID: \"9050192a-0ec2-45b9-8b2c-2beb0501b09b\") " pod="openshift-dns/node-resolver-vq2rj" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908912 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a882e61f-2b83-4ff4-8370-cba401a6d9c8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908952 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/026b5dc8-bcff-4afc-a784-baf95c1cec27-tmp\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908975 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-z8p96\" (UID: \"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908997 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0327c758-d1f5-481d-9cf2-96e4b5f27872-metrics-client-ca\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.909001 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/026b5dc8-bcff-4afc-a784-baf95c1cec27-tmp\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.908990 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3a695777-f42a-4cb6-8a31-c07c8daf321a-node-exporter-tls\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.909022 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-socket-dir\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.909085 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3a695777-f42a-4cb6-8a31-c07c8daf321a-node-exporter-textfile\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.909097 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-grpc-tls\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.909119 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3a695777-f42a-4cb6-8a31-c07c8daf321a-node-exporter-textfile\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.909118 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bee40e70-1cd3-4b89-8808-3abf1dcfbee9-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-n8k2s\" (UID: \"bee40e70-1cd3-4b89-8808-3abf1dcfbee9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-n8k2s" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.909148 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-lib-modules\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.909166 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0327c758-d1f5-481d-9cf2-96e4b5f27872-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.909182 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a2dc043c-52c0-4da0-8370-ed975ebb987f-ovnkube-script-lib\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.909209 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cba96b6a-ccc8-4c1c-b371-1bb5ece29afc-konnectivity-ca\") pod \"konnectivity-agent-ksg72\" (UID: \"cba96b6a-ccc8-4c1c-b371-1bb5ece29afc\") " pod="kube-system/konnectivity-agent-ksg72" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.909229 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15a5eea7-15b5-45a6-838b-35914cb59be1-tmp\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.909229 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-z8p96\" (UID: \"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.909247 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebf82283-d275-4640-bc44-5a461e4f9f9d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.909264 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0327c758-d1f5-481d-9cf2-96e4b5f27872-metrics-client-ca\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.909319 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bee40e70-1cd3-4b89-8808-3abf1dcfbee9-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-n8k2s\" (UID: \"bee40e70-1cd3-4b89-8808-3abf1dcfbee9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-n8k2s" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.909378 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15a5eea7-15b5-45a6-838b-35914cb59be1-tmp\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.909386 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbsdk\" (UniqueName: \"kubernetes.io/projected/ebf82283-d275-4640-bc44-5a461e4f9f9d-kube-api-access-pbsdk\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.909425 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-trusted-ca-bundle\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.909436 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a2dc043c-52c0-4da0-8370-ed975ebb987f-ovnkube-script-lib\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.909456 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c77a4d5b-a0d3-468b-aa66-7d607794aba1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-zbcg6\" (UID: \"c77a4d5b-a0d3-468b-aa66-7d607794aba1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.909473 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebf82283-d275-4640-bc44-5a461e4f9f9d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.909485 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc6n7\" (UniqueName: \"kubernetes.io/projected/05683773-8c40-4d7c-9cc9-ee9d71b25b23-kube-api-access-wc6n7\") pod \"dns-default-7cvm2\" (UID: \"05683773-8c40-4d7c-9cc9-ee9d71b25b23\") " pod="openshift-dns/dns-default-7cvm2" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:26.909493 5615 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.909512 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cba96b6a-ccc8-4c1c-b371-1bb5ece29afc-konnectivity-ca\") pod \"konnectivity-agent-ksg72\" (UID: \"cba96b6a-ccc8-4c1c-b371-1bb5ece29afc\") " pod="kube-system/konnectivity-agent-ksg72" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.909518 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.909471 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0327c758-d1f5-481d-9cf2-96e4b5f27872-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.909548 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-var-lib-openvswitch\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:26.909558 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-trusted-ca-bundle podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:27.40954343 +0000 UTC m=+1.877322133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-trusted-ca-bundle") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.909685 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15a5eea7-15b5-45a6-838b-35914cb59be1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:26.910124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.909693 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c77a4d5b-a0d3-468b-aa66-7d607794aba1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-zbcg6\" (UID: \"c77a4d5b-a0d3-468b-aa66-7d607794aba1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6" Nov 03 20:48:26.960149 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:26.960110 5615 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Nov 03 20:48:26.960149 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:26.960137 5615 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Nov 03 20:48:26.960149 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:26.960148 5615 projected.go:196] Error preparing data for projected volume kube-api-access-vw4cq for pod openshift-console/console-7f847c9697-nsrdm: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 20:48:26.960438 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:26.960227 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/195d22be-670c-4866-a303-1333085f8a67-kube-api-access-vw4cq podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:27.460212649 +0000 UTC m=+1.927991363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vw4cq" (UniqueName: "kubernetes.io/projected/195d22be-670c-4866-a303-1333085f8a67-kube-api-access-vw4cq") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 20:48:26.960754 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.960730 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbhtz\" (UniqueName: \"kubernetes.io/projected/bee40e70-1cd3-4b89-8808-3abf1dcfbee9-kube-api-access-rbhtz\") pod \"monitoring-plugin-77f4f76594-n8k2s\" (UID: \"bee40e70-1cd3-4b89-8808-3abf1dcfbee9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-n8k2s" Nov 03 20:48:26.964176 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.964153 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhkvl\" (UniqueName: \"kubernetes.io/projected/3a695777-f42a-4cb6-8a31-c07c8daf321a-kube-api-access-lhkvl\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:26.964369 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.964350 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5mgg\" (UniqueName: \"kubernetes.io/projected/955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81-kube-api-access-j5mgg\") pod \"network-check-target-lm85p\" (UID: \"955dbaeb-db8a-4a1a-ba32-2ee7e8a7ad81\") " pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:48:26.964684 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.964665 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6lpt\" (UniqueName: \"kubernetes.io/projected/026b5dc8-bcff-4afc-a784-baf95c1cec27-kube-api-access-p6lpt\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:26.964738 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.964667 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42v78\" (UniqueName: \"kubernetes.io/projected/a2dc043c-52c0-4da0-8370-ed975ebb987f-kube-api-access-42v78\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:26.964872 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.964856 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv6zc\" (UniqueName: \"kubernetes.io/projected/9050192a-0ec2-45b9-8b2c-2beb0501b09b-kube-api-access-tv6zc\") pod \"node-resolver-vq2rj\" (UID: \"9050192a-0ec2-45b9-8b2c-2beb0501b09b\") " pod="openshift-dns/node-resolver-vq2rj" Nov 03 20:48:26.965045 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.965031 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k85jh\" (UniqueName: \"kubernetes.io/projected/fac5da49-6649-4779-9529-089804436313-kube-api-access-k85jh\") pod \"ingress-canary-7vv52\" (UID: \"fac5da49-6649-4779-9529-089804436313\") " pod="openshift-ingress-canary/ingress-canary-7vv52" Nov 03 20:48:26.966288 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.966262 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prrnl\" (UniqueName: \"kubernetes.io/projected/a3e654a3-7d76-4793-9627-6334ba6db6d9-kube-api-access-prrnl\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:48:26.966288 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.966275 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkxvg\" (UniqueName: \"kubernetes.io/projected/eaf3e4d3-7019-42e5-9e1e-2927b7f22304-kube-api-access-nkxvg\") pod \"iptables-alerter-9bt29\" (UID: \"eaf3e4d3-7019-42e5-9e1e-2927b7f22304\") " pod="openshift-network-operator/iptables-alerter-9bt29" Nov 03 20:48:26.966715 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.966696 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pgk2\" (UniqueName: \"kubernetes.io/projected/c77a4d5b-a0d3-468b-aa66-7d607794aba1-kube-api-access-9pgk2\") pod \"openshift-state-metrics-7bd7489957-zbcg6\" (UID: \"c77a4d5b-a0d3-468b-aa66-7d607794aba1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zbcg6" Nov 03 20:48:26.966887 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.966869 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzx44\" (UniqueName: \"kubernetes.io/projected/e64dc44d-5470-4abb-8155-fdf7f24c253b-kube-api-access-zzx44\") pod \"insights-runtime-extractor-tms4q\" (UID: \"e64dc44d-5470-4abb-8155-fdf7f24c253b\") " pod="openshift-insights/insights-runtime-extractor-tms4q" Nov 03 20:48:26.966982 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.966960 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh5qw\" (UniqueName: \"kubernetes.io/projected/c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c-kube-api-access-dh5qw\") pod \"network-metrics-daemon-tqhnr\" (UID: \"c4ff8dbb-c78e-4bed-9f92-279c34c0ee9c\") " pod="openshift-multus/network-metrics-daemon-tqhnr" Nov 03 20:48:26.967033 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.966986 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ttzr\" (UniqueName: \"kubernetes.io/projected/a882e61f-2b83-4ff4-8370-cba401a6d9c8-kube-api-access-6ttzr\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:48:26.967257 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.967238 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbsdk\" (UniqueName: \"kubernetes.io/projected/ebf82283-d275-4640-bc44-5a461e4f9f9d-kube-api-access-pbsdk\") pod \"metrics-server-b9b549fcc-4cxct\" (UID: \"ebf82283-d275-4640-bc44-5a461e4f9f9d\") " pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:26.969215 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.969195 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc6n7\" (UniqueName: \"kubernetes.io/projected/05683773-8c40-4d7c-9cc9-ee9d71b25b23-kube-api-access-wc6n7\") pod \"dns-default-7cvm2\" (UID: \"05683773-8c40-4d7c-9cc9-ee9d71b25b23\") " pod="openshift-dns/dns-default-7cvm2" Nov 03 20:48:26.969288 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.969261 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbctt\" (UniqueName: \"kubernetes.io/projected/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-kube-api-access-cbctt\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:26.971742 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.971717 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf5tz\" (UniqueName: \"kubernetes.io/projected/bfca9b06-e0d2-4d0f-a75c-7827485cb42d-kube-api-access-tf5tz\") pod \"node-ca-lmq6t\" (UID: \"bfca9b06-e0d2-4d0f-a75c-7827485cb42d\") " pod="openshift-image-registry/node-ca-lmq6t" Nov 03 20:48:26.971835 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.971819 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7rds\" (UniqueName: \"kubernetes.io/projected/0327c758-d1f5-481d-9cf2-96e4b5f27872-kube-api-access-p7rds\") pod \"telemeter-client-5d484bf9b5-d4v5f\" (UID: \"0327c758-d1f5-481d-9cf2-96e4b5f27872\") " pod="openshift-monitoring/telemeter-client-5d484bf9b5-d4v5f" Nov 03 20:48:26.971942 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.971928 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6x9d\" (UniqueName: \"kubernetes.io/projected/15a5eea7-15b5-45a6-838b-35914cb59be1-kube-api-access-x6x9d\") pod \"thanos-querier-5d578896-ftx7j\" (UID: \"15a5eea7-15b5-45a6-838b-35914cb59be1\") " pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:26.973536 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:26.973515 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbvc2\" (UniqueName: \"kubernetes.io/projected/c14bf8f0-1335-4ee0-b49d-5d1f2836de1c-kube-api-access-fbvc2\") pod \"kube-state-metrics-5dc8d7d89b-z8p96\" (UID: \"c14bf8f0-1335-4ee0-b49d-5d1f2836de1c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z8p96" Nov 03 20:48:27.010488 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.010454 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eaf3e4d3-7019-42e5-9e1e-2927b7f22304-host-slash\") pod \"iptables-alerter-9bt29\" (UID: \"eaf3e4d3-7019-42e5-9e1e-2927b7f22304\") " pod="openshift-network-operator/iptables-alerter-9bt29" Nov 03 20:48:27.010488 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.010493 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3a695777-f42a-4cb6-8a31-c07c8daf321a-root\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:27.010700 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.010518 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-cni-netd\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:27.010700 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.010538 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-run\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:27.010700 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.010560 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-var-lib-kubelet\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:27.010700 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.010574 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eaf3e4d3-7019-42e5-9e1e-2927b7f22304-host-slash\") pod \"iptables-alerter-9bt29\" (UID: \"eaf3e4d3-7019-42e5-9e1e-2927b7f22304\") " pod="openshift-network-operator/iptables-alerter-9bt29" Nov 03 20:48:27.010700 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.010596 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-run-netns\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:27.010700 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.010599 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-cni-netd\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:27.010700 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.010618 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-run\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:27.010700 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.010579 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3a695777-f42a-4cb6-8a31-c07c8daf321a-root\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:27.010700 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.010640 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-run-netns\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:27.010700 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.010654 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-var-lib-kubelet\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:27.010700 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.010671 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-hostroot\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:27.010700 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.010701 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-run-ovn\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:27.011205 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.010731 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-etc-openvswitch\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:27.011205 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.010776 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-etc-openvswitch\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:27.011205 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.010775 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-hostroot\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:27.011205 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.010778 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-run-ovn\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:27.011205 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.010796 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-sysconfig\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:27.011205 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.010826 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-system-cni-dir\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:27.011205 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.010872 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-run-netns\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:27.011205 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.010876 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-system-cni-dir\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:27.011205 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.010889 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-node-log\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:27.011205 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.010905 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-multus-socket-dir-parent\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:27.011205 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.010925 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a882e61f-2b83-4ff4-8370-cba401a6d9c8-system-cni-dir\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:48:27.011205 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.010872 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-sysconfig\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:27.011205 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.010943 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-sysctl-d\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:27.011205 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.010965 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-run-netns\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:27.011205 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.010982 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-node-log\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:27.011205 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.010990 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-multus-socket-dir-parent\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:27.011205 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011013 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a882e61f-2b83-4ff4-8370-cba401a6d9c8-system-cni-dir\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:48:27.011205 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011029 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:27.011205 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011068 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-cni-bin\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:27.011205 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011110 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-var-lib-cni-multus\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:27.011205 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011118 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-sysctl-d\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:27.011205 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011116 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:27.011205 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011134 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-multus-conf-dir\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:27.011205 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011149 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-cni-bin\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:27.011205 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011162 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-var-lib-cni-multus\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:27.011205 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011165 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-sys-fs\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:48:27.011205 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011187 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/712fcecb-4757-4dd0-8f96-084b1e6ee903-kubelet-config\") pod \"global-pull-secret-syncer-mqmds\" (UID: \"712fcecb-4757-4dd0-8f96-084b1e6ee903\") " pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:48:27.011205 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011202 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-sys\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:27.011205 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011219 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-systemd-units\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:27.011205 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011188 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-multus-conf-dir\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011238 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/712fcecb-4757-4dd0-8f96-084b1e6ee903-kubelet-config\") pod \"global-pull-secret-syncer-mqmds\" (UID: \"712fcecb-4757-4dd0-8f96-084b1e6ee903\") " pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011244 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-cnibin\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011278 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-sys-fs\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011284 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-systemd-units\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011290 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-cnibin\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011284 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-run-systemd\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011286 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-sys\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011312 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-run-systemd\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011320 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-host\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011351 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-run-k8s-cni-cncf-io\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011369 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-var-lib-kubelet\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011380 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-host\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011389 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-modprobe-d\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011406 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e64dc44d-5470-4abb-8155-fdf7f24c253b-crio-socket\") pod \"insights-runtime-extractor-tms4q\" (UID: \"e64dc44d-5470-4abb-8155-fdf7f24c253b\") " pod="openshift-insights/insights-runtime-extractor-tms4q" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011410 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-run-k8s-cni-cncf-io\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011445 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-var-lib-cni-bin\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011459 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e64dc44d-5470-4abb-8155-fdf7f24c253b-crio-socket\") pod \"insights-runtime-extractor-tms4q\" (UID: \"e64dc44d-5470-4abb-8155-fdf7f24c253b\") " pod="openshift-insights/insights-runtime-extractor-tms4q" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011467 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-var-lib-kubelet\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011471 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011502 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-var-lib-cni-bin\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011504 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-systemd\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011510 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011518 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-modprobe-d\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011532 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-run-ovn-kubernetes\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011550 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a882e61f-2b83-4ff4-8370-cba401a6d9c8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011567 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-socket-dir\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011591 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-lib-modules\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011604 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-run-ovn-kubernetes\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011616 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a882e61f-2b83-4ff4-8370-cba401a6d9c8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011636 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-var-lib-openvswitch\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011632 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-socket-dir\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011660 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a695777-f42a-4cb6-8a31-c07c8daf321a-sys\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011690 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9050192a-0ec2-45b9-8b2c-2beb0501b09b-hosts-file\") pod \"node-resolver-vq2rj\" (UID: \"9050192a-0ec2-45b9-8b2c-2beb0501b09b\") " pod="openshift-dns/node-resolver-vq2rj" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011697 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a695777-f42a-4cb6-8a31-c07c8daf321a-sys\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011701 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-var-lib-openvswitch\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011665 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-systemd\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011713 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-lib-modules\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011725 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-log-socket\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011754 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-log-socket\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011773 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-multus-cni-dir\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011795 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-run-openvswitch\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011783 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9050192a-0ec2-45b9-8b2c-2beb0501b09b-hosts-file\") pod \"node-resolver-vq2rj\" (UID: \"9050192a-0ec2-45b9-8b2c-2beb0501b09b\") " pod="openshift-dns/node-resolver-vq2rj" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011814 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-kubelet\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011840 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-kubelet\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011846 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-slash\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011859 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-run-openvswitch\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011868 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-multus-cni-dir\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011880 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2dc043c-52c0-4da0-8370-ed975ebb987f-host-slash\") pod \"ovnkube-node-mfbd2\" (UID: \"a2dc043c-52c0-4da0-8370-ed975ebb987f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011889 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-registration-dir\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011907 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011923 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-run-multus-certs\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011928 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-registration-dir\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011946 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/712fcecb-4757-4dd0-8f96-084b1e6ee903-dbus\") pod \"global-pull-secret-syncer-mqmds\" (UID: \"712fcecb-4757-4dd0-8f96-084b1e6ee903\") " pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011974 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-device-dir\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011977 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.011968 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-host-run-multus-certs\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.012008 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3a695777-f42a-4cb6-8a31-c07c8daf321a-node-exporter-wtmp\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.012035 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-sysctl-conf\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.012067 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bfca9b06-e0d2-4d0f-a75c-7827485cb42d-host\") pod \"node-ca-lmq6t\" (UID: \"bfca9b06-e0d2-4d0f-a75c-7827485cb42d\") " pod="openshift-image-registry/node-ca-lmq6t" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.012094 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3a695777-f42a-4cb6-8a31-c07c8daf321a-node-exporter-wtmp\") pod \"node-exporter-ffhc8\" (UID: \"3a695777-f42a-4cb6-8a31-c07c8daf321a\") " pod="openshift-monitoring/node-exporter-ffhc8" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.012036 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a3e654a3-7d76-4793-9627-6334ba6db6d9-device-dir\") pod \"aws-ebs-csi-driver-node-xmszc\" (UID: \"a3e654a3-7d76-4793-9627-6334ba6db6d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmszc" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.012112 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-kubernetes\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.012142 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-etc-kubernetes\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.012146 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bfca9b06-e0d2-4d0f-a75c-7827485cb42d-host\") pod \"node-ca-lmq6t\" (UID: \"bfca9b06-e0d2-4d0f-a75c-7827485cb42d\") " pod="openshift-image-registry/node-ca-lmq6t" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.012154 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-kubernetes\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.012161 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a882e61f-2b83-4ff4-8370-cba401a6d9c8-cnibin\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.012186 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a882e61f-2b83-4ff4-8370-cba401a6d9c8-cnibin\") pod \"multus-additional-cni-plugins-5p62l\" (UID: \"a882e61f-2b83-4ff4-8370-cba401a6d9c8\") " pod="openshift-multus/multus-additional-cni-plugins-5p62l" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.012036 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/712fcecb-4757-4dd0-8f96-084b1e6ee903-dbus\") pod \"global-pull-secret-syncer-mqmds\" (UID: \"712fcecb-4757-4dd0-8f96-084b1e6ee903\") " pod="kube-system/global-pull-secret-syncer-mqmds" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.012194 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afd8ecac-e0b7-4453-a4b5-c17ba6504cbb-etc-kubernetes\") pod \"multus-thxwg\" (UID: \"afd8ecac-e0b7-4453-a4b5-c17ba6504cbb\") " pod="openshift-multus/multus-thxwg" Nov 03 20:48:27.012428 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.012202 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/026b5dc8-bcff-4afc-a784-baf95c1cec27-etc-sysctl-conf\") pod \"tuned-qwqjp\" (UID: \"026b5dc8-bcff-4afc-a784-baf95c1cec27\") " pod="openshift-cluster-node-tuning-operator/tuned-qwqjp" Nov 03 20:48:27.414984 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.414945 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-trusted-ca-bundle\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:27.415181 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.415015 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-service-ca\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:27.415181 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.415046 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-oauth-config\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:27.415181 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:27.415109 5615 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 20:48:27.415181 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:27.415146 5615 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Nov 03 20:48:27.415181 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:27.415154 5615 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Nov 03 20:48:27.415181 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:27.415183 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-trusted-ca-bundle podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:28.415163939 +0000 UTC m=+2.882942659 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-trusted-ca-bundle") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 20:48:27.415429 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.415114 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-console-config\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:27.415429 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:27.415191 5615 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Nov 03 20:48:27.415429 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:27.415206 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-service-ca podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:28.415189703 +0000 UTC m=+2.882968419 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-service-ca") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"service-ca" not registered Nov 03 20:48:27.415429 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:27.415228 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-console-config podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:28.415222749 +0000 UTC m=+2.883001453 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-console-config") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"console-config" not registered Nov 03 20:48:27.415429 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:27.415241 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-oauth-config podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:28.415234899 +0000 UTC m=+2.883013600 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-oauth-config") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"console-oauth-config" not registered Nov 03 20:48:27.415429 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.415272 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-serving-cert\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:27.415429 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.415293 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-oauth-serving-cert\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:27.415429 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:27.415345 5615 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Nov 03 20:48:27.415429 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:27.415367 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-oauth-serving-cert podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:28.415360897 +0000 UTC m=+2.883139610 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-oauth-serving-cert") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"oauth-serving-cert" not registered Nov 03 20:48:27.415429 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:27.415415 5615 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Nov 03 20:48:27.415702 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:27.415466 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-serving-cert podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:28.415451193 +0000 UTC m=+2.883229911 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-serving-cert") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"console-serving-cert" not registered Nov 03 20:48:27.515930 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:27.515892 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw4cq\" (UniqueName: \"kubernetes.io/projected/195d22be-670c-4866-a303-1333085f8a67-kube-api-access-vw4cq\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:27.516104 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:27.516026 5615 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Nov 03 20:48:27.516104 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:27.516040 5615 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Nov 03 20:48:27.516104 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:27.516050 5615 projected.go:196] Error preparing data for projected volume kube-api-access-vw4cq for pod openshift-console/console-7f847c9697-nsrdm: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 20:48:27.516233 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:27.516120 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/195d22be-670c-4866-a303-1333085f8a67-kube-api-access-vw4cq podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:28.516106881 +0000 UTC m=+2.983885596 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vw4cq" (UniqueName: "kubernetes.io/projected/195d22be-670c-4866-a303-1333085f8a67-kube-api-access-vw4cq") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 20:48:28.424193 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:28.424149 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-oauth-serving-cert\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:28.424193 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:28.424201 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-trusted-ca-bundle\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:28.424666 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:28.424245 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-service-ca\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:28.424666 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:28.424271 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-oauth-config\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:28.424666 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:28.424271 5615 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Nov 03 20:48:28.424666 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:28.424296 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-console-config\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:28.424666 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:28.424334 5615 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 20:48:28.424666 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:28.424321 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-serving-cert\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:28.424666 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:28.424354 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-oauth-serving-cert podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:30.424329635 +0000 UTC m=+4.892108349 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-oauth-serving-cert") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"oauth-serving-cert" not registered Nov 03 20:48:28.424666 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:28.424380 5615 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Nov 03 20:48:28.424666 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:28.424354 5615 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Nov 03 20:48:28.424666 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:28.424400 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-trusted-ca-bundle podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:30.424384446 +0000 UTC m=+4.892163161 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-trusted-ca-bundle") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 20:48:28.424666 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:28.424414 5615 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Nov 03 20:48:28.424666 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:28.424426 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-console-config podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:30.424412629 +0000 UTC m=+4.892191349 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-console-config") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"console-config" not registered Nov 03 20:48:28.424666 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:28.424424 5615 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Nov 03 20:48:28.424666 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:28.424445 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-service-ca podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:30.424436994 +0000 UTC m=+4.892215700 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-service-ca") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"service-ca" not registered Nov 03 20:48:28.424666 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:28.424467 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-serving-cert podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:30.424457848 +0000 UTC m=+4.892236566 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-serving-cert") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"console-serving-cert" not registered Nov 03 20:48:28.424666 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:28.424490 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-oauth-config podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:30.4244744 +0000 UTC m=+4.892253102 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-oauth-config") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"console-oauth-config" not registered Nov 03 20:48:28.525800 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:28.525761 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw4cq\" (UniqueName: \"kubernetes.io/projected/195d22be-670c-4866-a303-1333085f8a67-kube-api-access-vw4cq\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:28.526004 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:28.525926 5615 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Nov 03 20:48:28.526004 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:28.525945 5615 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Nov 03 20:48:28.526004 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:28.525956 5615 projected.go:196] Error preparing data for projected volume kube-api-access-vw4cq for pod openshift-console/console-7f847c9697-nsrdm: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 20:48:28.526138 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:28.526009 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/195d22be-670c-4866-a303-1333085f8a67-kube-api-access-vw4cq podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:30.525994816 +0000 UTC m=+4.993773523 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vw4cq" (UniqueName: "kubernetes.io/projected/195d22be-670c-4866-a303-1333085f8a67-kube-api-access-vw4cq") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 20:48:29.591055 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:29.591023 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7cvm2" Nov 03 20:48:30.307438 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:30.307409 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:30.443915 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:30.443856 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-console-config\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:30.443915 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:30.443923 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-serving-cert\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:30.444148 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:30.443945 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-oauth-serving-cert\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:30.444148 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:30.443970 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-trusted-ca-bundle\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:30.444148 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:30.443978 5615 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Nov 03 20:48:30.444148 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:30.444040 5615 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 20:48:30.444148 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:30.444056 5615 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Nov 03 20:48:30.444148 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:30.444103 5615 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Nov 03 20:48:30.444148 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:30.444059 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-console-config podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:34.444042785 +0000 UTC m=+8.911821499 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-console-config") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"console-config" not registered Nov 03 20:48:30.444449 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:30.444166 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-trusted-ca-bundle podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:34.444150863 +0000 UTC m=+8.911929567 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-trusted-ca-bundle") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 20:48:30.444449 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:30.444186 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-service-ca\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:30.444449 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:30.444223 5615 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Nov 03 20:48:30.444449 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:30.444225 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-oauth-config\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:30.444449 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:30.444267 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-oauth-serving-cert podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:34.444241615 +0000 UTC m=+8.912020322 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-oauth-serving-cert") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"oauth-serving-cert" not registered Nov 03 20:48:30.444449 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:30.444278 5615 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Nov 03 20:48:30.444449 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:30.444296 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-serving-cert podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:34.444280897 +0000 UTC m=+8.912059598 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-serving-cert") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"console-serving-cert" not registered Nov 03 20:48:30.444449 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:30.444315 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-service-ca podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:34.444306007 +0000 UTC m=+8.912084712 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-service-ca") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"service-ca" not registered Nov 03 20:48:30.444449 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:30.444335 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-oauth-config podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:34.444324244 +0000 UTC m=+8.912102949 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-oauth-config") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"console-oauth-config" not registered Nov 03 20:48:30.545276 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:30.545238 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw4cq\" (UniqueName: \"kubernetes.io/projected/195d22be-670c-4866-a303-1333085f8a67-kube-api-access-vw4cq\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:30.545455 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:30.545432 5615 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Nov 03 20:48:30.545551 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:30.545464 5615 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Nov 03 20:48:30.545551 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:30.545478 5615 projected.go:196] Error preparing data for projected volume kube-api-access-vw4cq for pod openshift-console/console-7f847c9697-nsrdm: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 20:48:30.545627 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:30.545562 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/195d22be-670c-4866-a303-1333085f8a67-kube-api-access-vw4cq podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:34.545544237 +0000 UTC m=+9.013322941 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vw4cq" (UniqueName: "kubernetes.io/projected/195d22be-670c-4866-a303-1333085f8a67-kube-api-access-vw4cq") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 20:48:31.518121 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:31.518092 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5d578896-ftx7j" Nov 03 20:48:33.503123 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:33.503093 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-n8k2s" Nov 03 20:48:33.552667 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:33.552632 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:48:34.486014 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:34.485973 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-console-config\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:34.486014 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:34.486021 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-serving-cert\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:34.486542 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:34.486045 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-oauth-serving-cert\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:34.486542 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:34.486108 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-trusted-ca-bundle\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:34.486542 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:34.486133 5615 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Nov 03 20:48:34.486542 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:34.486168 5615 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Nov 03 20:48:34.486542 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:34.486177 5615 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Nov 03 20:48:34.486542 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:34.486216 5615 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Nov 03 20:48:34.486542 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:34.486178 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-service-ca\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:34.486542 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:34.486232 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-console-config podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:42.486208791 +0000 UTC m=+16.953987511 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-console-config") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"console-config" not registered Nov 03 20:48:34.486542 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:34.486258 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-serving-cert podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:42.486249698 +0000 UTC m=+16.954028407 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-serving-cert") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"console-serving-cert" not registered Nov 03 20:48:34.486542 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:34.486274 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-oauth-serving-cert podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:42.486267016 +0000 UTC m=+16.954045717 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-oauth-serving-cert") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"oauth-serving-cert" not registered Nov 03 20:48:34.486542 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:34.486282 5615 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 20:48:34.486542 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:34.486289 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-service-ca podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:42.4862821 +0000 UTC m=+16.954060801 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-service-ca") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"service-ca" not registered Nov 03 20:48:34.486542 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:34.486393 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-trusted-ca-bundle podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:42.486380493 +0000 UTC m=+16.954159194 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-trusted-ca-bundle") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 20:48:34.486542 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:34.486428 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-oauth-config\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:34.486542 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:34.486508 5615 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Nov 03 20:48:34.486542 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:34.486559 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-oauth-config podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:42.486547941 +0000 UTC m=+16.954326649 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-oauth-config") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"console-oauth-config" not registered Nov 03 20:48:34.587519 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:34.587475 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw4cq\" (UniqueName: \"kubernetes.io/projected/195d22be-670c-4866-a303-1333085f8a67-kube-api-access-vw4cq\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:34.587913 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:34.587642 5615 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Nov 03 20:48:34.587913 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:34.587659 5615 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Nov 03 20:48:34.587913 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:34.587670 5615 projected.go:196] Error preparing data for projected volume kube-api-access-vw4cq for pod openshift-console/console-7f847c9697-nsrdm: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 20:48:34.587913 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:34.587723 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/195d22be-670c-4866-a303-1333085f8a67-kube-api-access-vw4cq podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:42.587710402 +0000 UTC m=+17.055489102 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vw4cq" (UniqueName: "kubernetes.io/projected/195d22be-670c-4866-a303-1333085f8a67-kube-api-access-vw4cq") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 20:48:36.051054 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:36.050981 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:36.051442 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:36.051257 5615 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:48:40.617369 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:40.617330 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 20:48:42.561757 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:42.561723 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-service-ca\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:42.561757 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:42.561765 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-oauth-config\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:42.562261 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:42.561789 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-console-config\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:42.562261 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:42.561863 5615 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Nov 03 20:48:42.562261 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:42.561893 5615 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Nov 03 20:48:42.562261 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:42.561910 5615 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Nov 03 20:48:42.562261 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:42.561943 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-console-config podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:58.561923988 +0000 UTC m=+33.029702709 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-console-config") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"console-config" not registered Nov 03 20:48:42.562261 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:42.561939 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-serving-cert\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:42.562261 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:42.561967 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-oauth-config podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:58.561954339 +0000 UTC m=+33.029733053 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-oauth-config") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"console-oauth-config" not registered Nov 03 20:48:42.562261 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:42.561987 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-service-ca podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:58.561978423 +0000 UTC m=+33.029757126 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-service-ca") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"service-ca" not registered Nov 03 20:48:42.562261 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:42.562026 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-oauth-serving-cert\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:42.562261 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:42.562033 5615 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Nov 03 20:48:42.562261 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:42.562088 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-trusted-ca-bundle\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:42.562261 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:42.562137 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-serving-cert podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:58.562120311 +0000 UTC m=+33.029899019 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-serving-cert") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"console-serving-cert" not registered Nov 03 20:48:42.562261 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:42.562146 5615 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Nov 03 20:48:42.562261 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:42.562155 5615 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 20:48:42.562261 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:42.562197 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-trusted-ca-bundle podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:58.562186238 +0000 UTC m=+33.029964959 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-trusted-ca-bundle") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 20:48:42.562261 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:42.562213 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-oauth-serving-cert podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:58.562203851 +0000 UTC m=+33.029982564 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-oauth-serving-cert") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : object "openshift-console"/"oauth-serving-cert" not registered Nov 03 20:48:42.662961 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:42.662923 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw4cq\" (UniqueName: \"kubernetes.io/projected/195d22be-670c-4866-a303-1333085f8a67-kube-api-access-vw4cq\") pod \"console-7f847c9697-nsrdm\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:42.663152 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:42.663116 5615 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Nov 03 20:48:42.663152 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:42.663154 5615 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Nov 03 20:48:42.663254 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:42.663166 5615 projected.go:196] Error preparing data for projected volume kube-api-access-vw4cq for pod openshift-console/console-7f847c9697-nsrdm: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 20:48:42.663254 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:42.663222 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/195d22be-670c-4866-a303-1333085f8a67-kube-api-access-vw4cq podName:195d22be-670c-4866-a303-1333085f8a67 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:58.663205972 +0000 UTC m=+33.130984691 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vw4cq" (UniqueName: "kubernetes.io/projected/195d22be-670c-4866-a303-1333085f8a67-kube-api-access-vw4cq") pod "console-7f847c9697-nsrdm" (UID: "195d22be-670c-4866-a303-1333085f8a67") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 20:48:45.587262 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:45.587231 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ksg72" Nov 03 20:48:48.478029 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:48.477993 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mfbd2" Nov 03 20:48:49.229241 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:49.229207 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-b9b549fcc-4cxct" Nov 03 20:48:51.772737 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:51.772695 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7f847c9697-nsrdm" podUID="195d22be-670c-4866-a303-1333085f8a67" containerName="console" containerID="cri-o://bac540ca14ec158883fadfe47d0dc4e0e5bfbd2b22dc73bb9cfe9cf7e95659d5" gracePeriod=15 Nov 03 20:48:52.013740 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.013713 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:52.153275 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.153205 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-console-config\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-console-config\") pod \"195d22be-670c-4866-a303-1333085f8a67\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " Nov 03 20:48:52.153275 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.153256 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/195d22be-670c-4866-a303-1333085f8a67-kube-api-access-vw4cq\" (UniqueName: \"kubernetes.io/projected/195d22be-670c-4866-a303-1333085f8a67-kube-api-access-vw4cq\") pod \"195d22be-670c-4866-a303-1333085f8a67\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " Nov 03 20:48:52.153494 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.153289 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-trusted-ca-bundle\") pod \"195d22be-670c-4866-a303-1333085f8a67\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " Nov 03 20:48:52.153494 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.153314 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/195d22be-670c-4866-a303-1333085f8a67-tmp\" (UniqueName: \"kubernetes.io/empty-dir/195d22be-670c-4866-a303-1333085f8a67-tmp\") pod \"195d22be-670c-4866-a303-1333085f8a67\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " Nov 03 20:48:52.153494 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.153343 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-serving-cert\") pod \"195d22be-670c-4866-a303-1333085f8a67\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " Nov 03 20:48:52.153494 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.153368 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-service-ca\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-service-ca\") pod \"195d22be-670c-4866-a303-1333085f8a67\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " Nov 03 20:48:52.153494 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.153393 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-oauth-config\") pod \"195d22be-670c-4866-a303-1333085f8a67\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " Nov 03 20:48:52.153494 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.153445 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-oauth-serving-cert\") pod \"195d22be-670c-4866-a303-1333085f8a67\" (UID: \"195d22be-670c-4866-a303-1333085f8a67\") " Nov 03 20:48:52.153785 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.153629 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-console-config" pod "195d22be-670c-4866-a303-1333085f8a67" (UID: "195d22be-670c-4866-a303-1333085f8a67"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:48:52.153785 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.153686 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195d22be-670c-4866-a303-1333085f8a67-tmp" pod "195d22be-670c-4866-a303-1333085f8a67" (UID: "195d22be-670c-4866-a303-1333085f8a67"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:48:52.153895 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.153841 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-trusted-ca-bundle" pod "195d22be-670c-4866-a303-1333085f8a67" (UID: "195d22be-670c-4866-a303-1333085f8a67"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:48:52.153945 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.153918 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-oauth-serving-cert" pod "195d22be-670c-4866-a303-1333085f8a67" (UID: "195d22be-670c-4866-a303-1333085f8a67"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:48:52.154092 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.154051 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-service-ca" pod "195d22be-670c-4866-a303-1333085f8a67" (UID: "195d22be-670c-4866-a303-1333085f8a67"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:48:52.155787 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.155751 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-serving-cert" pod "195d22be-670c-4866-a303-1333085f8a67" (UID: "195d22be-670c-4866-a303-1333085f8a67"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:48:52.155882 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.155815 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195d22be-670c-4866-a303-1333085f8a67-kube-api-access-vw4cq" pod "195d22be-670c-4866-a303-1333085f8a67" (UID: "195d22be-670c-4866-a303-1333085f8a67"). InnerVolumeSpecName "kube-api-access-vw4cq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:48:52.155882 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.155821 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-oauth-config" pod "195d22be-670c-4866-a303-1333085f8a67" (UID: "195d22be-670c-4866-a303-1333085f8a67"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:48:52.254971 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.254929 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw4cq\" (UniqueName: \"kubernetes.io/projected/195d22be-670c-4866-a303-1333085f8a67-kube-api-access-vw4cq\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:52.254971 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.254964 5615 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-trusted-ca-bundle\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:52.254971 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.254976 5615 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/195d22be-670c-4866-a303-1333085f8a67-tmp\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:52.254971 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.254984 5615 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-serving-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:52.255261 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.254993 5615 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:52.255261 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.255002 5615 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/195d22be-670c-4866-a303-1333085f8a67-console-oauth-config\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:52.255261 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.255013 5615 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-oauth-serving-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:52.255261 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.255023 5615 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/195d22be-670c-4866-a303-1333085f8a67-console-config\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:52.544549 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.544515 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c67f9676-hg9l2"] Nov 03 20:48:52.544808 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.544796 5615 cpu_manager.go:414] "containerMap: removing container" podUID="195d22be-670c-4866-a303-1333085f8a67" containerName="console" Nov 03 20:48:52.544856 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.544809 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="195d22be-670c-4866-a303-1333085f8a67" containerName="console" Nov 03 20:48:52.544889 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.544883 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="195d22be-670c-4866-a303-1333085f8a67" containerName="console" Nov 03 20:48:52.547708 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.547685 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c67f9676-hg9l2" Nov 03 20:48:52.604828 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.604795 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 20:48:52.605066 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.605045 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 20:48:52.605201 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.605103 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 20:48:52.605201 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.605108 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 20:48:52.605201 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.605127 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-znh84\"" Nov 03 20:48:52.652183 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.652153 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 20:48:52.652183 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.652157 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 20:48:52.652395 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.652230 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 20:48:52.657069 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.657046 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1d947d5-9112-4762-aab0-54b7213d2951-console-oauth-config\") pod \"console-5c67f9676-hg9l2\" (UID: \"b1d947d5-9112-4762-aab0-54b7213d2951\") " pod="openshift-console/console-5c67f9676-hg9l2" Nov 03 20:48:52.657191 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.657088 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1d947d5-9112-4762-aab0-54b7213d2951-console-config\") pod \"console-5c67f9676-hg9l2\" (UID: \"b1d947d5-9112-4762-aab0-54b7213d2951\") " pod="openshift-console/console-5c67f9676-hg9l2" Nov 03 20:48:52.657191 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.657110 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1d947d5-9112-4762-aab0-54b7213d2951-tmp\") pod \"console-5c67f9676-hg9l2\" (UID: \"b1d947d5-9112-4762-aab0-54b7213d2951\") " pod="openshift-console/console-5c67f9676-hg9l2" Nov 03 20:48:52.657191 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.657167 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1d947d5-9112-4762-aab0-54b7213d2951-trusted-ca-bundle\") pod \"console-5c67f9676-hg9l2\" (UID: \"b1d947d5-9112-4762-aab0-54b7213d2951\") " pod="openshift-console/console-5c67f9676-hg9l2" Nov 03 20:48:52.657283 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.657246 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1d947d5-9112-4762-aab0-54b7213d2951-service-ca\") pod \"console-5c67f9676-hg9l2\" (UID: \"b1d947d5-9112-4762-aab0-54b7213d2951\") " pod="openshift-console/console-5c67f9676-hg9l2" Nov 03 20:48:52.657283 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.657275 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1d947d5-9112-4762-aab0-54b7213d2951-console-serving-cert\") pod \"console-5c67f9676-hg9l2\" (UID: \"b1d947d5-9112-4762-aab0-54b7213d2951\") " pod="openshift-console/console-5c67f9676-hg9l2" Nov 03 20:48:52.657346 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.657296 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srv46\" (UniqueName: \"kubernetes.io/projected/b1d947d5-9112-4762-aab0-54b7213d2951-kube-api-access-srv46\") pod \"console-5c67f9676-hg9l2\" (UID: \"b1d947d5-9112-4762-aab0-54b7213d2951\") " pod="openshift-console/console-5c67f9676-hg9l2" Nov 03 20:48:52.657434 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.657419 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1d947d5-9112-4762-aab0-54b7213d2951-oauth-serving-cert\") pod \"console-5c67f9676-hg9l2\" (UID: \"b1d947d5-9112-4762-aab0-54b7213d2951\") " pod="openshift-console/console-5c67f9676-hg9l2" Nov 03 20:48:52.747446 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.747408 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 20:48:52.747862 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.747840 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c67f9676-hg9l2"] Nov 03 20:48:52.758669 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.758648 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1d947d5-9112-4762-aab0-54b7213d2951-console-serving-cert\") pod \"console-5c67f9676-hg9l2\" (UID: \"b1d947d5-9112-4762-aab0-54b7213d2951\") " pod="openshift-console/console-5c67f9676-hg9l2" Nov 03 20:48:52.758727 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.758676 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srv46\" (UniqueName: \"kubernetes.io/projected/b1d947d5-9112-4762-aab0-54b7213d2951-kube-api-access-srv46\") pod \"console-5c67f9676-hg9l2\" (UID: \"b1d947d5-9112-4762-aab0-54b7213d2951\") " pod="openshift-console/console-5c67f9676-hg9l2" Nov 03 20:48:52.758727 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.758708 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1d947d5-9112-4762-aab0-54b7213d2951-oauth-serving-cert\") pod \"console-5c67f9676-hg9l2\" (UID: \"b1d947d5-9112-4762-aab0-54b7213d2951\") " pod="openshift-console/console-5c67f9676-hg9l2" Nov 03 20:48:52.758822 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.758728 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1d947d5-9112-4762-aab0-54b7213d2951-console-oauth-config\") pod \"console-5c67f9676-hg9l2\" (UID: \"b1d947d5-9112-4762-aab0-54b7213d2951\") " pod="openshift-console/console-5c67f9676-hg9l2" Nov 03 20:48:52.758822 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.758746 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1d947d5-9112-4762-aab0-54b7213d2951-console-config\") pod \"console-5c67f9676-hg9l2\" (UID: \"b1d947d5-9112-4762-aab0-54b7213d2951\") " pod="openshift-console/console-5c67f9676-hg9l2" Nov 03 20:48:52.758822 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.758766 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1d947d5-9112-4762-aab0-54b7213d2951-tmp\") pod \"console-5c67f9676-hg9l2\" (UID: \"b1d947d5-9112-4762-aab0-54b7213d2951\") " pod="openshift-console/console-5c67f9676-hg9l2" Nov 03 20:48:52.758822 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.758793 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1d947d5-9112-4762-aab0-54b7213d2951-trusted-ca-bundle\") pod \"console-5c67f9676-hg9l2\" (UID: \"b1d947d5-9112-4762-aab0-54b7213d2951\") " pod="openshift-console/console-5c67f9676-hg9l2" Nov 03 20:48:52.758822 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.758820 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1d947d5-9112-4762-aab0-54b7213d2951-service-ca\") pod \"console-5c67f9676-hg9l2\" (UID: \"b1d947d5-9112-4762-aab0-54b7213d2951\") " pod="openshift-console/console-5c67f9676-hg9l2" Nov 03 20:48:52.759154 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.759108 5615 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 20:48:52.759312 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.759290 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1d947d5-9112-4762-aab0-54b7213d2951-tmp\") pod \"console-5c67f9676-hg9l2\" (UID: \"b1d947d5-9112-4762-aab0-54b7213d2951\") " pod="openshift-console/console-5c67f9676-hg9l2" Nov 03 20:48:52.759555 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.759534 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1d947d5-9112-4762-aab0-54b7213d2951-console-config\") pod \"console-5c67f9676-hg9l2\" (UID: \"b1d947d5-9112-4762-aab0-54b7213d2951\") " pod="openshift-console/console-5c67f9676-hg9l2" Nov 03 20:48:52.759627 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.759601 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1d947d5-9112-4762-aab0-54b7213d2951-oauth-serving-cert\") pod \"console-5c67f9676-hg9l2\" (UID: \"b1d947d5-9112-4762-aab0-54b7213d2951\") " pod="openshift-console/console-5c67f9676-hg9l2" Nov 03 20:48:52.759679 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.759653 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1d947d5-9112-4762-aab0-54b7213d2951-service-ca\") pod \"console-5c67f9676-hg9l2\" (UID: \"b1d947d5-9112-4762-aab0-54b7213d2951\") " pod="openshift-console/console-5c67f9676-hg9l2" Nov 03 20:48:52.759771 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.759750 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1d947d5-9112-4762-aab0-54b7213d2951-trusted-ca-bundle\") pod \"console-5c67f9676-hg9l2\" (UID: \"b1d947d5-9112-4762-aab0-54b7213d2951\") " pod="openshift-console/console-5c67f9676-hg9l2" Nov 03 20:48:52.761392 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.761372 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1d947d5-9112-4762-aab0-54b7213d2951-console-oauth-config\") pod \"console-5c67f9676-hg9l2\" (UID: \"b1d947d5-9112-4762-aab0-54b7213d2951\") " pod="openshift-console/console-5c67f9676-hg9l2" Nov 03 20:48:52.761466 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.761453 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1d947d5-9112-4762-aab0-54b7213d2951-console-serving-cert\") pod \"console-5c67f9676-hg9l2\" (UID: \"b1d947d5-9112-4762-aab0-54b7213d2951\") " pod="openshift-console/console-5c67f9676-hg9l2" Nov 03 20:48:52.798558 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.798485 5615 generic.go:356] "Generic (PLEG): container finished" podID="195d22be-670c-4866-a303-1333085f8a67" containerID="bac540ca14ec158883fadfe47d0dc4e0e5bfbd2b22dc73bb9cfe9cf7e95659d5" exitCode=0 Nov 03 20:48:52.798558 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.798539 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f847c9697-nsrdm" Nov 03 20:48:52.798558 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.798553 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f847c9697-nsrdm" event={"ID":"195d22be-670c-4866-a303-1333085f8a67","Type":"ContainerDied","Data":"bac540ca14ec158883fadfe47d0dc4e0e5bfbd2b22dc73bb9cfe9cf7e95659d5"} Nov 03 20:48:52.798987 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.798602 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f847c9697-nsrdm" event={"ID":"195d22be-670c-4866-a303-1333085f8a67","Type":"ContainerDied","Data":"e666b74d740ff3bc6f845fa19ccf562c1e0ee6a6f6475572c685ac7d72b6fbb3"} Nov 03 20:48:52.798987 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.798629 5615 scope.go:122] "RemoveContainer" containerID="bac540ca14ec158883fadfe47d0dc4e0e5bfbd2b22dc73bb9cfe9cf7e95659d5" Nov 03 20:48:52.806652 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.806633 5615 scope.go:122] "RemoveContainer" containerID="bac540ca14ec158883fadfe47d0dc4e0e5bfbd2b22dc73bb9cfe9cf7e95659d5" Nov 03 20:48:52.806921 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:48:52.806903 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bac540ca14ec158883fadfe47d0dc4e0e5bfbd2b22dc73bb9cfe9cf7e95659d5\": container with ID starting with bac540ca14ec158883fadfe47d0dc4e0e5bfbd2b22dc73bb9cfe9cf7e95659d5 not found: ID does not exist" containerID="bac540ca14ec158883fadfe47d0dc4e0e5bfbd2b22dc73bb9cfe9cf7e95659d5" Nov 03 20:48:52.807009 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.806930 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bac540ca14ec158883fadfe47d0dc4e0e5bfbd2b22dc73bb9cfe9cf7e95659d5"} err="failed to get container status \"bac540ca14ec158883fadfe47d0dc4e0e5bfbd2b22dc73bb9cfe9cf7e95659d5\": rpc error: code = NotFound desc = could not find container \"bac540ca14ec158883fadfe47d0dc4e0e5bfbd2b22dc73bb9cfe9cf7e95659d5\": container with ID starting with bac540ca14ec158883fadfe47d0dc4e0e5bfbd2b22dc73bb9cfe9cf7e95659d5 not found: ID does not exist" Nov 03 20:48:52.824912 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.824882 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srv46\" (UniqueName: \"kubernetes.io/projected/b1d947d5-9112-4762-aab0-54b7213d2951-kube-api-access-srv46\") pod \"console-5c67f9676-hg9l2\" (UID: \"b1d947d5-9112-4762-aab0-54b7213d2951\") " pod="openshift-console/console-5c67f9676-hg9l2" Nov 03 20:48:52.858310 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.858274 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c67f9676-hg9l2" Nov 03 20:48:52.916675 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.916623 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f847c9697-nsrdm"] Nov 03 20:48:52.920543 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:52.920519 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f847c9697-nsrdm"] Nov 03 20:48:53.221172 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:53.221069 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c67f9676-hg9l2"] Nov 03 20:48:53.224426 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:48:53.224396 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1d947d5_9112_4762_aab0_54b7213d2951.slice/crio-4fe2395eab8ead4ba5a13803bfa0e1afa4f5197206f66366fae9a208742a2a8a WatchSource:0}: Error finding container 4fe2395eab8ead4ba5a13803bfa0e1afa4f5197206f66366fae9a208742a2a8a: Status 404 returned error can't find the container with id 4fe2395eab8ead4ba5a13803bfa0e1afa4f5197206f66366fae9a208742a2a8a Nov 03 20:48:53.648858 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:53.648822 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="195d22be-670c-4866-a303-1333085f8a67" path="/var/lib/kubelet/pods/195d22be-670c-4866-a303-1333085f8a67/volumes" Nov 03 20:48:53.804721 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:53.804679 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c67f9676-hg9l2" event={"ID":"b1d947d5-9112-4762-aab0-54b7213d2951","Type":"ContainerStarted","Data":"be93af1b11a5c651a5f38141becc39b52c2184b68598137c565d67935161fd8e"} Nov 03 20:48:53.805182 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:53.804743 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c67f9676-hg9l2" event={"ID":"b1d947d5-9112-4762-aab0-54b7213d2951","Type":"ContainerStarted","Data":"4fe2395eab8ead4ba5a13803bfa0e1afa4f5197206f66366fae9a208742a2a8a"} Nov 03 20:48:53.918645 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:48:53.918554 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5c67f9676-hg9l2" podStartSLOduration=1.918540162 podStartE2EDuration="1.918540162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:53.91831088 +0000 UTC m=+28.386089603" watchObservedRunningTime="2026-11-03 20:48:53.918540162 +0000 UTC m=+28.386318884" Nov 03 20:49:01.006944 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.006912 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 20:49:01.010892 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.010872 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.024758 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.024729 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1ab03d50-196f-497f-bd8c-381588ef9237-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.024963 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.024764 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1ab03d50-196f-497f-bd8c-381588ef9237-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.024963 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.024783 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1ab03d50-196f-497f-bd8c-381588ef9237-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.024963 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.024940 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1ab03d50-196f-497f-bd8c-381588ef9237-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.025135 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.024986 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1ab03d50-196f-497f-bd8c-381588ef9237-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.025135 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.025013 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1ab03d50-196f-497f-bd8c-381588ef9237-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.025210 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.025156 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1ab03d50-196f-497f-bd8c-381588ef9237-config-out\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.025210 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.025179 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1ab03d50-196f-497f-bd8c-381588ef9237-config-volume\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.025210 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.025198 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ab03d50-196f-497f-bd8c-381588ef9237-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.025361 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.025224 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1ab03d50-196f-497f-bd8c-381588ef9237-web-config\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.025361 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.025279 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ab03d50-196f-497f-bd8c-381588ef9237-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.025361 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.025316 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grsnl\" (UniqueName: \"kubernetes.io/projected/1ab03d50-196f-497f-bd8c-381588ef9237-kube-api-access-grsnl\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.025476 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.025392 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1ab03d50-196f-497f-bd8c-381588ef9237-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.126305 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.126258 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1ab03d50-196f-497f-bd8c-381588ef9237-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.126305 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.126307 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1ab03d50-196f-497f-bd8c-381588ef9237-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.126575 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.126325 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1ab03d50-196f-497f-bd8c-381588ef9237-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.126575 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.126377 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1ab03d50-196f-497f-bd8c-381588ef9237-config-out\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.126575 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.126412 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1ab03d50-196f-497f-bd8c-381588ef9237-config-volume\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.126575 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.126442 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ab03d50-196f-497f-bd8c-381588ef9237-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.126575 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.126473 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1ab03d50-196f-497f-bd8c-381588ef9237-web-config\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.126575 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.126545 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ab03d50-196f-497f-bd8c-381588ef9237-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.126575 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.126576 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grsnl\" (UniqueName: \"kubernetes.io/projected/1ab03d50-196f-497f-bd8c-381588ef9237-kube-api-access-grsnl\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.126859 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.126598 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1ab03d50-196f-497f-bd8c-381588ef9237-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.126859 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.126631 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1ab03d50-196f-497f-bd8c-381588ef9237-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.126859 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.126660 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1ab03d50-196f-497f-bd8c-381588ef9237-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.126859 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.126688 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1ab03d50-196f-497f-bd8c-381588ef9237-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.127007 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.126963 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1ab03d50-196f-497f-bd8c-381588ef9237-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.127258 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.127234 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ab03d50-196f-497f-bd8c-381588ef9237-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.128590 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.128571 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1ab03d50-196f-497f-bd8c-381588ef9237-config-out\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.158150 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.158109 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 20:49:01.158350 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.158195 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 20:49:01.158350 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.158222 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 20:49:01.158350 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.158228 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 20:49:01.158350 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.158232 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 20:49:01.158350 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.158268 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 20:49:01.158350 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.158256 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 20:49:01.158350 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.158304 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 20:49:01.158698 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.158370 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-swkc2\"" Nov 03 20:49:01.158698 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.158410 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 20:49:01.159289 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.159267 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 20:49:01.168547 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.168517 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ab03d50-196f-497f-bd8c-381588ef9237-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.170501 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.170474 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1ab03d50-196f-497f-bd8c-381588ef9237-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.170653 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.170615 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1ab03d50-196f-497f-bd8c-381588ef9237-web-config\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.170798 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.170775 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1ab03d50-196f-497f-bd8c-381588ef9237-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.170874 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.170775 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1ab03d50-196f-497f-bd8c-381588ef9237-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.170874 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.170825 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1ab03d50-196f-497f-bd8c-381588ef9237-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.170987 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.170947 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1ab03d50-196f-497f-bd8c-381588ef9237-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.171165 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.171144 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1ab03d50-196f-497f-bd8c-381588ef9237-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.171348 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.171331 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1ab03d50-196f-497f-bd8c-381588ef9237-config-volume\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.185093 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.185052 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grsnl\" (UniqueName: \"kubernetes.io/projected/1ab03d50-196f-497f-bd8c-381588ef9237-kube-api-access-grsnl\") pod \"alertmanager-main-0\" (UID: \"1ab03d50-196f-497f-bd8c-381588ef9237\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.331488 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.331387 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:49:01.797394 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.797370 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 20:49:01.799884 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:49:01.799857 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab03d50_196f_497f_bd8c_381588ef9237.slice/crio-8d61a4c9e65d0feee9b56be3be1d360bfdcf54459bff2cf57fe7d784e2f0d017 WatchSource:0}: Error finding container 8d61a4c9e65d0feee9b56be3be1d360bfdcf54459bff2cf57fe7d784e2f0d017: Status 404 returned error can't find the container with id 8d61a4c9e65d0feee9b56be3be1d360bfdcf54459bff2cf57fe7d784e2f0d017 Nov 03 20:49:01.834241 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:01.834212 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ab03d50-196f-497f-bd8c-381588ef9237","Type":"ContainerStarted","Data":"8d61a4c9e65d0feee9b56be3be1d360bfdcf54459bff2cf57fe7d784e2f0d017"} Nov 03 20:49:02.841188 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:02.841154 5615 generic.go:356] "Generic (PLEG): container finished" podID="1ab03d50-196f-497f-bd8c-381588ef9237" containerID="e839ead34180e3046b764f99f4030791aebc0b87e49ad8ff980358668fd47632" exitCode=0 Nov 03 20:49:02.841615 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:02.841229 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ab03d50-196f-497f-bd8c-381588ef9237","Type":"ContainerDied","Data":"e839ead34180e3046b764f99f4030791aebc0b87e49ad8ff980358668fd47632"} Nov 03 20:49:02.842629 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:02.842612 5615 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:49:02.858763 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:02.858726 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c67f9676-hg9l2" Nov 03 20:49:02.858919 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:02.858792 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c67f9676-hg9l2" Nov 03 20:49:02.863266 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:02.863244 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c67f9676-hg9l2" Nov 03 20:49:03.556238 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:03.556207 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lm85p" Nov 03 20:49:03.849387 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:03.849315 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c67f9676-hg9l2" Nov 03 20:49:07.863686 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:07.863651 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ab03d50-196f-497f-bd8c-381588ef9237","Type":"ContainerStarted","Data":"13da91431845f0136a5547c5555d4b88c9038368f89bbae65df4f11e703f952c"} Nov 03 20:49:07.864120 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:07.863716 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ab03d50-196f-497f-bd8c-381588ef9237","Type":"ContainerStarted","Data":"20b1336578ed256ab5ee0b26f8fff4b04694dbd44817956bd2cd1200a4b53c7c"} Nov 03 20:49:07.864120 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:07.863743 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ab03d50-196f-497f-bd8c-381588ef9237","Type":"ContainerStarted","Data":"7f6b324d51d44d9196b1ef2461521fceb36c8dda51b231b071671a0feee7c9bc"} Nov 03 20:49:07.864120 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:07.863768 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ab03d50-196f-497f-bd8c-381588ef9237","Type":"ContainerStarted","Data":"03701666a14e475e1f485107dabba94a2d5b6425ffac57fd2366f9f2451eb60e"} Nov 03 20:49:07.864120 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:07.863793 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ab03d50-196f-497f-bd8c-381588ef9237","Type":"ContainerStarted","Data":"21ac5b13a853d11d928e9c222ac038acef3a1196eccefc2630f07219f95fbd64"} Nov 03 20:49:07.864120 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:07.863815 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ab03d50-196f-497f-bd8c-381588ef9237","Type":"ContainerStarted","Data":"6697e4241fdd384088d181cfd83b0a5e264ef52581c83831950c4c2e4e11e63a"} Nov 03 20:49:08.130016 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:49:08.129918 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.751178449 podStartE2EDuration="8.129904923s" totalImagesPullingTime="4.369787132s" totalInitContainerRuntime="8.939342ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:49:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:49:08.082773707 +0000 UTC m=+42.550552468" watchObservedRunningTime="2026-11-03 20:49:08.129904923 +0000 UTC m=+42.597683636" Nov 03 20:49:10.617187 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:49:10.617151 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 20:49:40.617931 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:49:40.617890 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 20:50:10.617446 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:50:10.617356 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 20:50:40.617547 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:50:40.617503 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 20:51:10.616861 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:51:10.616819 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 20:51:40.617492 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:51:40.617404 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 20:52:10.617404 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:52:10.617359 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 20:52:16.652537 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.652502 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qmvdd"] Nov 03 20:52:16.655388 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.655370 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qmvdd" Nov 03 20:52:16.675933 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.675905 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d22c0a0b-69de-469c-9c6b-16125658dbfb-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qmvdd\" (UID: \"d22c0a0b-69de-469c-9c6b-16125658dbfb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qmvdd" Nov 03 20:52:16.676024 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.675945 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d22c0a0b-69de-469c-9c6b-16125658dbfb-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qmvdd\" (UID: \"d22c0a0b-69de-469c-9c6b-16125658dbfb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qmvdd" Nov 03 20:52:16.676024 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.675969 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfcss\" (UniqueName: \"kubernetes.io/projected/d22c0a0b-69de-469c-9c6b-16125658dbfb-kube-api-access-lfcss\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qmvdd\" (UID: \"d22c0a0b-69de-469c-9c6b-16125658dbfb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qmvdd" Nov 03 20:52:16.705252 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.705225 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-7sxz2\"" Nov 03 20:52:16.752519 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.752477 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 20:52:16.752677 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.752539 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 20:52:16.753378 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.753358 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qmvdd"] Nov 03 20:52:16.777259 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.777222 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d22c0a0b-69de-469c-9c6b-16125658dbfb-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qmvdd\" (UID: \"d22c0a0b-69de-469c-9c6b-16125658dbfb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qmvdd" Nov 03 20:52:16.777439 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.777266 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d22c0a0b-69de-469c-9c6b-16125658dbfb-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qmvdd\" (UID: \"d22c0a0b-69de-469c-9c6b-16125658dbfb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qmvdd" Nov 03 20:52:16.777439 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.777290 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfcss\" (UniqueName: \"kubernetes.io/projected/d22c0a0b-69de-469c-9c6b-16125658dbfb-kube-api-access-lfcss\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qmvdd\" (UID: \"d22c0a0b-69de-469c-9c6b-16125658dbfb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qmvdd" Nov 03 20:52:16.777674 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.777651 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d22c0a0b-69de-469c-9c6b-16125658dbfb-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qmvdd\" (UID: \"d22c0a0b-69de-469c-9c6b-16125658dbfb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qmvdd" Nov 03 20:52:16.777711 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.777681 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d22c0a0b-69de-469c-9c6b-16125658dbfb-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qmvdd\" (UID: \"d22c0a0b-69de-469c-9c6b-16125658dbfb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qmvdd" Nov 03 20:52:16.836361 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.836323 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfcss\" (UniqueName: \"kubernetes.io/projected/d22c0a0b-69de-469c-9c6b-16125658dbfb-kube-api-access-lfcss\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qmvdd\" (UID: \"d22c0a0b-69de-469c-9c6b-16125658dbfb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qmvdd" Nov 03 20:52:16.883169 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.883134 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz"] Nov 03 20:52:16.885467 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.885449 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" Nov 03 20:52:16.935391 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.935293 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 20:52:16.935582 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.935484 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 20:52:16.935582 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.935489 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 20:52:16.935582 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.935528 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 20:52:16.935582 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.935555 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 20:52:16.935582 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.935568 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-bh5bp\"" Nov 03 20:52:16.935759 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.935599 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 20:52:16.946060 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.945979 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz"] Nov 03 20:52:16.968328 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.968297 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qmvdd" Nov 03 20:52:16.979024 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.978992 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzf2m\" (UniqueName: \"kubernetes.io/projected/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-kube-api-access-bzf2m\") pod \"istiod-openshift-gateway-7876bfcb74-ltmzz\" (UID: \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" Nov 03 20:52:16.979125 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.979062 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-ltmzz\" (UID: \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" Nov 03 20:52:16.979200 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.979178 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-ltmzz\" (UID: \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" Nov 03 20:52:16.979267 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.979250 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-ltmzz\" (UID: \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" Nov 03 20:52:16.979311 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.979293 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-ltmzz\" (UID: \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" Nov 03 20:52:16.979350 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.979334 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-ltmzz\" (UID: \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" Nov 03 20:52:16.979385 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:16.979367 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-ltmzz\" (UID: \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" Nov 03 20:52:17.079878 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:17.079831 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzf2m\" (UniqueName: \"kubernetes.io/projected/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-kube-api-access-bzf2m\") pod \"istiod-openshift-gateway-7876bfcb74-ltmzz\" (UID: \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" Nov 03 20:52:17.080063 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:17.079888 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-ltmzz\" (UID: \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" Nov 03 20:52:17.080063 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:17.079924 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-ltmzz\" (UID: \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" Nov 03 20:52:17.080063 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:17.079952 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-ltmzz\" (UID: \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" Nov 03 20:52:17.080063 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:17.079975 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-ltmzz\" (UID: \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" Nov 03 20:52:17.080063 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:17.080006 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-ltmzz\" (UID: \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" Nov 03 20:52:17.080303 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:17.080140 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-ltmzz\" (UID: \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" Nov 03 20:52:17.080803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:17.080773 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-ltmzz\" (UID: \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" Nov 03 20:52:17.082590 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:17.082558 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-ltmzz\" (UID: \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" Nov 03 20:52:17.082590 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:17.082581 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-ltmzz\" (UID: \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" Nov 03 20:52:17.082892 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:17.082870 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-ltmzz\" (UID: \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" Nov 03 20:52:17.082957 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:17.082942 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-ltmzz\" (UID: \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" Nov 03 20:52:17.138494 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:17.138458 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-ltmzz\" (UID: \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" Nov 03 20:52:17.138688 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:17.138669 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzf2m\" (UniqueName: \"kubernetes.io/projected/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-kube-api-access-bzf2m\") pod \"istiod-openshift-gateway-7876bfcb74-ltmzz\" (UID: \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" Nov 03 20:52:17.196395 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:17.196311 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" Nov 03 20:52:17.331655 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:17.331620 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qmvdd"] Nov 03 20:52:17.334683 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:52:17.334647 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd22c0a0b_69de_469c_9c6b_16125658dbfb.slice/crio-db6ae382232b8b77dc0fad3641faa9aa8a79c29148e64c4ce81013e9e31aaf40 WatchSource:0}: Error finding container db6ae382232b8b77dc0fad3641faa9aa8a79c29148e64c4ce81013e9e31aaf40: Status 404 returned error can't find the container with id db6ae382232b8b77dc0fad3641faa9aa8a79c29148e64c4ce81013e9e31aaf40 Nov 03 20:52:17.526946 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:17.526909 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qmvdd" event={"ID":"d22c0a0b-69de-469c-9c6b-16125658dbfb","Type":"ContainerStarted","Data":"db6ae382232b8b77dc0fad3641faa9aa8a79c29148e64c4ce81013e9e31aaf40"} Nov 03 20:52:17.562596 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:17.562565 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz"] Nov 03 20:52:17.565305 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:52:17.565275 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ffea0c2_79af_4d03_9948_6bc6f1223e6a.slice/crio-1cfda76d90b597a0c1ec226cf280c249477e58a74286a9ff97f65f6250da4f82 WatchSource:0}: Error finding container 1cfda76d90b597a0c1ec226cf280c249477e58a74286a9ff97f65f6250da4f82: Status 404 returned error can't find the container with id 1cfda76d90b597a0c1ec226cf280c249477e58a74286a9ff97f65f6250da4f82 Nov 03 20:52:18.532890 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:18.532846 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" event={"ID":"4ffea0c2-79af-4d03-9948-6bc6f1223e6a","Type":"ContainerStarted","Data":"1cfda76d90b597a0c1ec226cf280c249477e58a74286a9ff97f65f6250da4f82"} Nov 03 20:52:21.446381 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:21.446349 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:52:21.446646 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:21.446414 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:52:21.549625 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:21.549575 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" event={"ID":"4ffea0c2-79af-4d03-9948-6bc6f1223e6a","Type":"ContainerStarted","Data":"d2b1e207290d200a63a37896d325a811391a9ffda2460f62c56c244f859e4a66"} Nov 03 20:52:21.549793 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:21.549641 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" Nov 03 20:52:21.667282 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:21.667222 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" podStartSLOduration=1.789052656 podStartE2EDuration="5.667205871s" totalImagesPullingTime="3.878153215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:21.666500309 +0000 UTC m=+236.134279031" watchObservedRunningTime="2026-11-03 20:52:21.667205871 +0000 UTC m=+236.134984593" Nov 03 20:52:22.559648 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:22.558654 5615 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-ltmzz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:52:22.559648 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:22.558731 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" podUID="4ffea0c2-79af-4d03-9948-6bc6f1223e6a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:52:25.558214 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:25.558182 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" Nov 03 20:52:27.576441 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:27.576405 5615 generic.go:356] "Generic (PLEG): container finished" podID="d22c0a0b-69de-469c-9c6b-16125658dbfb" containerID="6bba74419ff888580c123794a1893fb21fbd865a477b58b315018ed4fa3320cd" exitCode=0 Nov 03 20:52:27.576841 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:27.576483 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qmvdd" event={"ID":"d22c0a0b-69de-469c-9c6b-16125658dbfb","Type":"ContainerDied","Data":"6bba74419ff888580c123794a1893fb21fbd865a477b58b315018ed4fa3320cd"} Nov 03 20:52:29.585670 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:29.585633 5615 generic.go:356] "Generic (PLEG): container finished" podID="d22c0a0b-69de-469c-9c6b-16125658dbfb" containerID="11fa049ed4a4e122aff6fab2d969351014c10bc8aa048477889cbdb2e6db2068" exitCode=0 Nov 03 20:52:29.586139 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:29.585727 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qmvdd" event={"ID":"d22c0a0b-69de-469c-9c6b-16125658dbfb","Type":"ContainerDied","Data":"11fa049ed4a4e122aff6fab2d969351014c10bc8aa048477889cbdb2e6db2068"} Nov 03 20:52:39.626994 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:39.626958 5615 generic.go:356] "Generic (PLEG): container finished" podID="d22c0a0b-69de-469c-9c6b-16125658dbfb" containerID="a2a89d961a4611689739d5f50deced25c17e1040b9f4e485d3d6263d45e53a5a" exitCode=0 Nov 03 20:52:39.627398 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:39.627019 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qmvdd" event={"ID":"d22c0a0b-69de-469c-9c6b-16125658dbfb","Type":"ContainerDied","Data":"a2a89d961a4611689739d5f50deced25c17e1040b9f4e485d3d6263d45e53a5a"} Nov 03 20:52:40.617145 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:52:40.617106 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 20:52:40.768113 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:40.768061 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qmvdd" Nov 03 20:52:40.903326 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:40.903237 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d22c0a0b-69de-469c-9c6b-16125658dbfb-kube-api-access-lfcss\" (UniqueName: \"kubernetes.io/projected/d22c0a0b-69de-469c-9c6b-16125658dbfb-kube-api-access-lfcss\") pod \"d22c0a0b-69de-469c-9c6b-16125658dbfb\" (UID: \"d22c0a0b-69de-469c-9c6b-16125658dbfb\") " Nov 03 20:52:40.903326 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:40.903295 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d22c0a0b-69de-469c-9c6b-16125658dbfb-util\" (UniqueName: \"kubernetes.io/empty-dir/d22c0a0b-69de-469c-9c6b-16125658dbfb-util\") pod \"d22c0a0b-69de-469c-9c6b-16125658dbfb\" (UID: \"d22c0a0b-69de-469c-9c6b-16125658dbfb\") " Nov 03 20:52:40.903547 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:40.903387 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d22c0a0b-69de-469c-9c6b-16125658dbfb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d22c0a0b-69de-469c-9c6b-16125658dbfb-bundle\") pod \"d22c0a0b-69de-469c-9c6b-16125658dbfb\" (UID: \"d22c0a0b-69de-469c-9c6b-16125658dbfb\") " Nov 03 20:52:40.903908 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:40.903881 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d22c0a0b-69de-469c-9c6b-16125658dbfb-bundle" pod "d22c0a0b-69de-469c-9c6b-16125658dbfb" (UID: "d22c0a0b-69de-469c-9c6b-16125658dbfb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:40.905523 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:40.905500 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d22c0a0b-69de-469c-9c6b-16125658dbfb-kube-api-access-lfcss" pod "d22c0a0b-69de-469c-9c6b-16125658dbfb" (UID: "d22c0a0b-69de-469c-9c6b-16125658dbfb"). InnerVolumeSpecName "kube-api-access-lfcss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:40.907429 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:40.907402 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d22c0a0b-69de-469c-9c6b-16125658dbfb-util" pod "d22c0a0b-69de-469c-9c6b-16125658dbfb" (UID: "d22c0a0b-69de-469c-9c6b-16125658dbfb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:41.004070 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:41.004032 5615 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d22c0a0b-69de-469c-9c6b-16125658dbfb-bundle\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:41.004070 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:41.004068 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfcss\" (UniqueName: \"kubernetes.io/projected/d22c0a0b-69de-469c-9c6b-16125658dbfb-kube-api-access-lfcss\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:41.004288 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:41.004108 5615 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d22c0a0b-69de-469c-9c6b-16125658dbfb-util\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:41.635204 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:41.635161 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qmvdd" event={"ID":"d22c0a0b-69de-469c-9c6b-16125658dbfb","Type":"ContainerDied","Data":"db6ae382232b8b77dc0fad3641faa9aa8a79c29148e64c4ce81013e9e31aaf40"} Nov 03 20:52:41.635370 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:41.635208 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qmvdd" Nov 03 20:52:41.635370 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:52:41.635254 5615 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db6ae382232b8b77dc0fad3641faa9aa8a79c29148e64c4ce81013e9e31aaf40" Nov 03 20:53:09.919176 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:09.919094 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-wwhrj"] Nov 03 20:53:09.919666 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:09.919425 5615 cpu_manager.go:414] "containerMap: removing container" podUID="d22c0a0b-69de-469c-9c6b-16125658dbfb" containerName="pull" Nov 03 20:53:09.919666 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:09.919437 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d22c0a0b-69de-469c-9c6b-16125658dbfb" containerName="pull" Nov 03 20:53:09.919666 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:09.919467 5615 cpu_manager.go:414] "containerMap: removing container" podUID="d22c0a0b-69de-469c-9c6b-16125658dbfb" containerName="util" Nov 03 20:53:09.919666 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:09.919472 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d22c0a0b-69de-469c-9c6b-16125658dbfb" containerName="util" Nov 03 20:53:09.919666 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:09.919479 5615 cpu_manager.go:414] "containerMap: removing container" podUID="d22c0a0b-69de-469c-9c6b-16125658dbfb" containerName="extract" Nov 03 20:53:09.919666 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:09.919485 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d22c0a0b-69de-469c-9c6b-16125658dbfb" containerName="extract" Nov 03 20:53:09.919666 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:09.919542 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="d22c0a0b-69de-469c-9c6b-16125658dbfb" containerName="extract" Nov 03 20:53:09.936471 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:09.936444 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-wwhrj" Nov 03 20:53:10.028186 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:10.028148 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96e80c27-c5d2-4b61-956a-234e95c174ab-tmp\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-wwhrj\" (UID: \"96e80c27-c5d2-4b61-956a-234e95c174ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-wwhrj" Nov 03 20:53:10.028186 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:10.028185 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/96e80c27-c5d2-4b61-956a-234e95c174ab-metrics-cert\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-wwhrj\" (UID: \"96e80c27-c5d2-4b61-956a-234e95c174ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-wwhrj" Nov 03 20:53:10.028432 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:10.028206 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/96e80c27-c5d2-4b61-956a-234e95c174ab-webhook-cert\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-wwhrj\" (UID: \"96e80c27-c5d2-4b61-956a-234e95c174ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-wwhrj" Nov 03 20:53:10.028432 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:10.028312 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/96e80c27-c5d2-4b61-956a-234e95c174ab-manifests\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-wwhrj\" (UID: \"96e80c27-c5d2-4b61-956a-234e95c174ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-wwhrj" Nov 03 20:53:10.028432 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:10.028338 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/96e80c27-c5d2-4b61-956a-234e95c174ab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-wwhrj\" (UID: \"96e80c27-c5d2-4b61-956a-234e95c174ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-wwhrj" Nov 03 20:53:10.028432 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:10.028356 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmtct\" (UniqueName: \"kubernetes.io/projected/96e80c27-c5d2-4b61-956a-234e95c174ab-kube-api-access-lmtct\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-wwhrj\" (UID: \"96e80c27-c5d2-4b61-956a-234e95c174ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-wwhrj" Nov 03 20:53:10.056861 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:10.056820 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-pc7cb\"" Nov 03 20:53:10.057022 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:10.056937 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-wwhrj"] Nov 03 20:53:10.057022 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:10.056994 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 20:53:10.057194 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:10.057180 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 20:53:10.057492 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:10.057458 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 20:53:10.057670 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:10.057574 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 20:53:10.057670 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:10.057590 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 20:53:10.129208 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:10.129173 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/96e80c27-c5d2-4b61-956a-234e95c174ab-manifests\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-wwhrj\" (UID: \"96e80c27-c5d2-4b61-956a-234e95c174ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-wwhrj" Nov 03 20:53:10.129379 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:10.129217 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/96e80c27-c5d2-4b61-956a-234e95c174ab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-wwhrj\" (UID: \"96e80c27-c5d2-4b61-956a-234e95c174ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-wwhrj" Nov 03 20:53:10.129379 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:10.129235 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmtct\" (UniqueName: \"kubernetes.io/projected/96e80c27-c5d2-4b61-956a-234e95c174ab-kube-api-access-lmtct\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-wwhrj\" (UID: \"96e80c27-c5d2-4b61-956a-234e95c174ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-wwhrj" Nov 03 20:53:10.129465 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:10.129444 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96e80c27-c5d2-4b61-956a-234e95c174ab-tmp\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-wwhrj\" (UID: \"96e80c27-c5d2-4b61-956a-234e95c174ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-wwhrj" Nov 03 20:53:10.129535 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:10.129490 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/96e80c27-c5d2-4b61-956a-234e95c174ab-metrics-cert\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-wwhrj\" (UID: \"96e80c27-c5d2-4b61-956a-234e95c174ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-wwhrj" Nov 03 20:53:10.129535 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:10.129519 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/96e80c27-c5d2-4b61-956a-234e95c174ab-webhook-cert\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-wwhrj\" (UID: \"96e80c27-c5d2-4b61-956a-234e95c174ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-wwhrj" Nov 03 20:53:10.129672 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:10.129650 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/96e80c27-c5d2-4b61-956a-234e95c174ab-manifests\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-wwhrj\" (UID: \"96e80c27-c5d2-4b61-956a-234e95c174ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-wwhrj" Nov 03 20:53:10.131738 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:10.131710 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/96e80c27-c5d2-4b61-956a-234e95c174ab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-wwhrj\" (UID: \"96e80c27-c5d2-4b61-956a-234e95c174ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-wwhrj" Nov 03 20:53:10.131854 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:10.131714 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96e80c27-c5d2-4b61-956a-234e95c174ab-tmp\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-wwhrj\" (UID: \"96e80c27-c5d2-4b61-956a-234e95c174ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-wwhrj" Nov 03 20:53:10.131938 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:10.131916 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/96e80c27-c5d2-4b61-956a-234e95c174ab-webhook-cert\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-wwhrj\" (UID: \"96e80c27-c5d2-4b61-956a-234e95c174ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-wwhrj" Nov 03 20:53:10.131974 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:10.131922 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/96e80c27-c5d2-4b61-956a-234e95c174ab-metrics-cert\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-wwhrj\" (UID: \"96e80c27-c5d2-4b61-956a-234e95c174ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-wwhrj" Nov 03 20:53:10.188032 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:10.187960 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmtct\" (UniqueName: \"kubernetes.io/projected/96e80c27-c5d2-4b61-956a-234e95c174ab-kube-api-access-lmtct\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-wwhrj\" (UID: \"96e80c27-c5d2-4b61-956a-234e95c174ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-wwhrj" Nov 03 20:53:10.277344 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:10.277300 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-wwhrj" Nov 03 20:53:10.618965 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:53:10.618926 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 20:53:10.810340 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:10.810308 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-wwhrj"] Nov 03 20:53:10.814948 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:53:10.814916 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e80c27_c5d2_4b61_956a_234e95c174ab.slice/crio-9e59aa0ac0ab71e0f42fd68d4aef160f7817bd5c1b36ec76cbcd4d808baef45b WatchSource:0}: Error finding container 9e59aa0ac0ab71e0f42fd68d4aef160f7817bd5c1b36ec76cbcd4d808baef45b: Status 404 returned error can't find the container with id 9e59aa0ac0ab71e0f42fd68d4aef160f7817bd5c1b36ec76cbcd4d808baef45b Nov 03 20:53:11.746830 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:11.746792 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-wwhrj" event={"ID":"96e80c27-c5d2-4b61-956a-234e95c174ab","Type":"ContainerStarted","Data":"9e59aa0ac0ab71e0f42fd68d4aef160f7817bd5c1b36ec76cbcd4d808baef45b"} Nov 03 20:53:14.761216 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:14.761181 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-wwhrj" event={"ID":"96e80c27-c5d2-4b61-956a-234e95c174ab","Type":"ContainerStarted","Data":"16b853db669c50db53d86402043bd19b40aed2c13a000f80638a7af121d7bb95"} Nov 03 20:53:15.766650 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:15.766613 5615 generic.go:356] "Generic (PLEG): container finished" podID="96e80c27-c5d2-4b61-956a-234e95c174ab" containerID="16b853db669c50db53d86402043bd19b40aed2c13a000f80638a7af121d7bb95" exitCode=0 Nov 03 20:53:15.767139 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:15.766701 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-wwhrj" event={"ID":"96e80c27-c5d2-4b61-956a-234e95c174ab","Type":"ContainerDied","Data":"16b853db669c50db53d86402043bd19b40aed2c13a000f80638a7af121d7bb95"} Nov 03 20:53:16.772788 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:16.772749 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-wwhrj" event={"ID":"96e80c27-c5d2-4b61-956a-234e95c174ab","Type":"ContainerStarted","Data":"51e0e75f133ebdf6e98ffafac0ca3923e087143c1abb05739320517acf729b2f"} Nov 03 20:53:16.773289 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:16.772872 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-wwhrj" Nov 03 20:53:16.999437 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:16.999375 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-wwhrj" podStartSLOduration=3.236878827 podStartE2EDuration="7.999359059s" totalImagesPullingTime="4.301853617s" totalInitContainerRuntime="460.626615ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:16.951840075 +0000 UTC m=+291.419618796" watchObservedRunningTime="2026-11-03 20:53:16.999359059 +0000 UTC m=+291.467137781" Nov 03 20:53:27.779332 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:27.779301 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-wwhrj" Nov 03 20:53:31.675262 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:31.675226 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-4r7zq"] Nov 03 20:53:31.680358 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:31.680337 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4r7zq" Nov 03 20:53:31.733996 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:31.733953 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-4r7zq"] Nov 03 20:53:31.780354 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:31.780319 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 20:53:31.780354 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:31.780339 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 20:53:31.804303 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:31.804264 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-gw8v6\"" Nov 03 20:53:31.825629 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:31.825593 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4bc3444-2d9a-4627-935a-9ee2276cfd18-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-4r7zq\" (UID: \"b4bc3444-2d9a-4627-935a-9ee2276cfd18\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4r7zq" Nov 03 20:53:31.825788 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:31.825634 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4trsj\" (UniqueName: \"kubernetes.io/projected/b4bc3444-2d9a-4627-935a-9ee2276cfd18-kube-api-access-4trsj\") pod \"cert-manager-cainjector-5c57bbd4fc-4r7zq\" (UID: \"b4bc3444-2d9a-4627-935a-9ee2276cfd18\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4r7zq" Nov 03 20:53:31.926701 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:31.926609 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4bc3444-2d9a-4627-935a-9ee2276cfd18-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-4r7zq\" (UID: \"b4bc3444-2d9a-4627-935a-9ee2276cfd18\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4r7zq" Nov 03 20:53:31.926701 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:31.926664 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4trsj\" (UniqueName: \"kubernetes.io/projected/b4bc3444-2d9a-4627-935a-9ee2276cfd18-kube-api-access-4trsj\") pod \"cert-manager-cainjector-5c57bbd4fc-4r7zq\" (UID: \"b4bc3444-2d9a-4627-935a-9ee2276cfd18\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4r7zq" Nov 03 20:53:31.985144 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:31.985112 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4bc3444-2d9a-4627-935a-9ee2276cfd18-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-4r7zq\" (UID: \"b4bc3444-2d9a-4627-935a-9ee2276cfd18\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4r7zq" Nov 03 20:53:31.985300 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:31.985174 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4trsj\" (UniqueName: \"kubernetes.io/projected/b4bc3444-2d9a-4627-935a-9ee2276cfd18-kube-api-access-4trsj\") pod \"cert-manager-cainjector-5c57bbd4fc-4r7zq\" (UID: \"b4bc3444-2d9a-4627-935a-9ee2276cfd18\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4r7zq" Nov 03 20:53:32.003907 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:32.003871 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4r7zq" Nov 03 20:53:32.374009 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:32.373964 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-4r7zq"] Nov 03 20:53:32.377114 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:53:32.377042 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4bc3444_2d9a_4627_935a_9ee2276cfd18.slice/crio-1f878164a0e72d865dc7f305f66c17ba4e3fde094297782f9112db7ec97eb6e1 WatchSource:0}: Error finding container 1f878164a0e72d865dc7f305f66c17ba4e3fde094297782f9112db7ec97eb6e1: Status 404 returned error can't find the container with id 1f878164a0e72d865dc7f305f66c17ba4e3fde094297782f9112db7ec97eb6e1 Nov 03 20:53:32.837264 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:32.837227 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4r7zq" event={"ID":"b4bc3444-2d9a-4627-935a-9ee2276cfd18","Type":"ContainerStarted","Data":"1f878164a0e72d865dc7f305f66c17ba4e3fde094297782f9112db7ec97eb6e1"} Nov 03 20:53:36.855663 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:36.855625 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4r7zq" event={"ID":"b4bc3444-2d9a-4627-935a-9ee2276cfd18","Type":"ContainerStarted","Data":"33d19acf019cd13e24dff70b99384bec5504f5863ae95d84c5adf9f90347a89f"} Nov 03 20:53:36.966468 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:36.966412 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4r7zq" podStartSLOduration=1.6217701930000001 podStartE2EDuration="5.966398635s" totalImagesPullingTime="4.344628442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:36.96574905 +0000 UTC m=+311.433527774" watchObservedRunningTime="2026-11-03 20:53:36.966398635 +0000 UTC m=+311.434177358" Nov 03 20:53:40.618240 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:53:40.618202 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 20:53:42.919736 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:42.919698 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-d69f97dbf-d25vp"] Nov 03 20:53:42.922214 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:42.922195 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d69f97dbf-d25vp" Nov 03 20:53:43.007008 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:43.006973 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d69f97dbf-d25vp"] Nov 03 20:53:43.038137 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:43.038097 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smbhr\" (UniqueName: \"kubernetes.io/projected/0fcbdbb2-d97d-4e8a-8f30-497ca0d41624-kube-api-access-smbhr\") pod \"kube-auth-proxy-d69f97dbf-d25vp\" (UID: \"0fcbdbb2-d97d-4e8a-8f30-497ca0d41624\") " pod="openshift-ingress/kube-auth-proxy-d69f97dbf-d25vp" Nov 03 20:53:43.038302 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:43.038166 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fcbdbb2-d97d-4e8a-8f30-497ca0d41624-tls-certs\") pod \"kube-auth-proxy-d69f97dbf-d25vp\" (UID: \"0fcbdbb2-d97d-4e8a-8f30-497ca0d41624\") " pod="openshift-ingress/kube-auth-proxy-d69f97dbf-d25vp" Nov 03 20:53:43.038302 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:43.038193 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fcbdbb2-d97d-4e8a-8f30-497ca0d41624-tmp\") pod \"kube-auth-proxy-d69f97dbf-d25vp\" (UID: \"0fcbdbb2-d97d-4e8a-8f30-497ca0d41624\") " pod="openshift-ingress/kube-auth-proxy-d69f97dbf-d25vp" Nov 03 20:53:43.056624 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:43.056587 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 20:53:43.082525 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:43.082491 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 20:53:43.082715 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:43.082554 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-64knl\"" Nov 03 20:53:43.139292 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:43.139251 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fcbdbb2-d97d-4e8a-8f30-497ca0d41624-tls-certs\") pod \"kube-auth-proxy-d69f97dbf-d25vp\" (UID: \"0fcbdbb2-d97d-4e8a-8f30-497ca0d41624\") " pod="openshift-ingress/kube-auth-proxy-d69f97dbf-d25vp" Nov 03 20:53:43.139292 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:43.139295 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fcbdbb2-d97d-4e8a-8f30-497ca0d41624-tmp\") pod \"kube-auth-proxy-d69f97dbf-d25vp\" (UID: \"0fcbdbb2-d97d-4e8a-8f30-497ca0d41624\") " pod="openshift-ingress/kube-auth-proxy-d69f97dbf-d25vp" Nov 03 20:53:43.139538 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:43.139356 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smbhr\" (UniqueName: \"kubernetes.io/projected/0fcbdbb2-d97d-4e8a-8f30-497ca0d41624-kube-api-access-smbhr\") pod \"kube-auth-proxy-d69f97dbf-d25vp\" (UID: \"0fcbdbb2-d97d-4e8a-8f30-497ca0d41624\") " pod="openshift-ingress/kube-auth-proxy-d69f97dbf-d25vp" Nov 03 20:53:43.141712 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:43.141680 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fcbdbb2-d97d-4e8a-8f30-497ca0d41624-tmp\") pod \"kube-auth-proxy-d69f97dbf-d25vp\" (UID: \"0fcbdbb2-d97d-4e8a-8f30-497ca0d41624\") " pod="openshift-ingress/kube-auth-proxy-d69f97dbf-d25vp" Nov 03 20:53:43.142024 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:43.141999 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fcbdbb2-d97d-4e8a-8f30-497ca0d41624-tls-certs\") pod \"kube-auth-proxy-d69f97dbf-d25vp\" (UID: \"0fcbdbb2-d97d-4e8a-8f30-497ca0d41624\") " pod="openshift-ingress/kube-auth-proxy-d69f97dbf-d25vp" Nov 03 20:53:43.198154 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:43.198085 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smbhr\" (UniqueName: \"kubernetes.io/projected/0fcbdbb2-d97d-4e8a-8f30-497ca0d41624-kube-api-access-smbhr\") pod \"kube-auth-proxy-d69f97dbf-d25vp\" (UID: \"0fcbdbb2-d97d-4e8a-8f30-497ca0d41624\") " pod="openshift-ingress/kube-auth-proxy-d69f97dbf-d25vp" Nov 03 20:53:43.238883 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:43.238850 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d69f97dbf-d25vp" Nov 03 20:53:43.600875 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:43.600685 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d69f97dbf-d25vp"] Nov 03 20:53:43.604241 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:53:43.604208 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fcbdbb2_d97d_4e8a_8f30_497ca0d41624.slice/crio-2d32df60dcf280931c12c8293350c19df38f956d47aa9cd86651092cf5cd1295 WatchSource:0}: Error finding container 2d32df60dcf280931c12c8293350c19df38f956d47aa9cd86651092cf5cd1295: Status 404 returned error can't find the container with id 2d32df60dcf280931c12c8293350c19df38f956d47aa9cd86651092cf5cd1295 Nov 03 20:53:43.885235 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:43.885153 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d69f97dbf-d25vp" event={"ID":"0fcbdbb2-d97d-4e8a-8f30-497ca0d41624","Type":"ContainerStarted","Data":"2d32df60dcf280931c12c8293350c19df38f956d47aa9cd86651092cf5cd1295"} Nov 03 20:53:46.902186 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:46.902097 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d69f97dbf-d25vp" event={"ID":"0fcbdbb2-d97d-4e8a-8f30-497ca0d41624","Type":"ContainerStarted","Data":"d617354c6e2f895838c195e39a709356cdddaae77fb90f8bac0bb94f6307b63e"} Nov 03 20:53:47.013133 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:47.013054 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-d69f97dbf-d25vp" podStartSLOduration=2.12350798 podStartE2EDuration="5.013041388s" totalImagesPullingTime="2.889533408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:47.011969589 +0000 UTC m=+321.479748312" watchObservedRunningTime="2026-11-03 20:53:47.013041388 +0000 UTC m=+321.480820110" Nov 03 20:53:48.790250 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:48.790214 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-snqk8"] Nov 03 20:53:48.796269 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:48.796247 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-snqk8" Nov 03 20:53:48.845862 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:48.845831 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-snqk8"] Nov 03 20:53:48.892675 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:48.892647 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 20:53:48.892842 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:48.892695 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dq8fb\"" Nov 03 20:53:48.894003 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:48.893974 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t2fc\" (UniqueName: \"kubernetes.io/projected/32026c4b-92f7-493a-8067-cabb159f7eed-kube-api-access-4t2fc\") pod \"authorino-56c6c9ddd5-snqk8\" (UID: \"32026c4b-92f7-493a-8067-cabb159f7eed\") " pod="kuadrant-system/authorino-56c6c9ddd5-snqk8" Nov 03 20:53:48.895444 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:48.895425 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 20:53:48.994444 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:48.994404 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t2fc\" (UniqueName: \"kubernetes.io/projected/32026c4b-92f7-493a-8067-cabb159f7eed-kube-api-access-4t2fc\") pod \"authorino-56c6c9ddd5-snqk8\" (UID: \"32026c4b-92f7-493a-8067-cabb159f7eed\") " pod="kuadrant-system/authorino-56c6c9ddd5-snqk8" Nov 03 20:53:49.052832 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:49.052762 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t2fc\" (UniqueName: \"kubernetes.io/projected/32026c4b-92f7-493a-8067-cabb159f7eed-kube-api-access-4t2fc\") pod \"authorino-56c6c9ddd5-snqk8\" (UID: \"32026c4b-92f7-493a-8067-cabb159f7eed\") " pod="kuadrant-system/authorino-56c6c9ddd5-snqk8" Nov 03 20:53:49.107180 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:49.107144 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-snqk8" Nov 03 20:53:49.469135 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:49.469024 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-snqk8"] Nov 03 20:53:49.472190 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:53:49.472151 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32026c4b_92f7_493a_8067_cabb159f7eed.slice/crio-8342f8620655108cbe04b97aad2d5179c2354675cb3f1789702a8493541802c1 WatchSource:0}: Error finding container 8342f8620655108cbe04b97aad2d5179c2354675cb3f1789702a8493541802c1: Status 404 returned error can't find the container with id 8342f8620655108cbe04b97aad2d5179c2354675cb3f1789702a8493541802c1 Nov 03 20:53:49.914653 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:49.914616 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-snqk8" event={"ID":"32026c4b-92f7-493a-8067-cabb159f7eed","Type":"ContainerStarted","Data":"8342f8620655108cbe04b97aad2d5179c2354675cb3f1789702a8493541802c1"} Nov 03 20:53:53.934069 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:53.934023 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-snqk8" event={"ID":"32026c4b-92f7-493a-8067-cabb159f7eed","Type":"ContainerStarted","Data":"9100a42a2fa93a4d086523a20bab0d350db337979d54d6771143342909dc07c4"} Nov 03 20:53:54.042701 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:54.042648 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-snqk8" podStartSLOduration=2.389195054 podStartE2EDuration="6.042636216s" totalImagesPullingTime="3.653441162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:54.042210214 +0000 UTC m=+328.509988938" watchObservedRunningTime="2026-11-03 20:53:54.042636216 +0000 UTC m=+328.510414941" Nov 03 20:53:56.009981 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:56.009945 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-9qt8v"] Nov 03 20:53:56.014480 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:56.014452 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9qt8v" Nov 03 20:53:56.057380 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:56.057342 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42cba095-3e85-4cd0-a6fa-255c8086733b-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-9qt8v\" (UID: \"42cba095-3e85-4cd0-a6fa-255c8086733b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9qt8v" Nov 03 20:53:56.057548 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:56.057395 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42cba095-3e85-4cd0-a6fa-255c8086733b-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-9qt8v\" (UID: \"42cba095-3e85-4cd0-a6fa-255c8086733b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9qt8v" Nov 03 20:53:56.057548 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:56.057483 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2xkr\" (UniqueName: \"kubernetes.io/projected/42cba095-3e85-4cd0-a6fa-255c8086733b-kube-api-access-d2xkr\") pod \"kserve-module-controller-manager-7566cf5fc8-9qt8v\" (UID: \"42cba095-3e85-4cd0-a6fa-255c8086733b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9qt8v" Nov 03 20:53:56.057548 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:56.057534 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/42cba095-3e85-4cd0-a6fa-255c8086733b-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-9qt8v\" (UID: \"42cba095-3e85-4cd0-a6fa-255c8086733b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9qt8v" Nov 03 20:53:56.064015 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:56.063984 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 20:53:56.064176 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:56.064062 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 20:53:56.111228 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:56.111185 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-6hzw4\"" Nov 03 20:53:56.158659 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:56.158621 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/42cba095-3e85-4cd0-a6fa-255c8086733b-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-9qt8v\" (UID: \"42cba095-3e85-4cd0-a6fa-255c8086733b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9qt8v" Nov 03 20:53:56.158866 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:56.158680 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42cba095-3e85-4cd0-a6fa-255c8086733b-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-9qt8v\" (UID: \"42cba095-3e85-4cd0-a6fa-255c8086733b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9qt8v" Nov 03 20:53:56.158866 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:56.158714 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42cba095-3e85-4cd0-a6fa-255c8086733b-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-9qt8v\" (UID: \"42cba095-3e85-4cd0-a6fa-255c8086733b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9qt8v" Nov 03 20:53:56.158866 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:56.158740 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2xkr\" (UniqueName: \"kubernetes.io/projected/42cba095-3e85-4cd0-a6fa-255c8086733b-kube-api-access-d2xkr\") pod \"kserve-module-controller-manager-7566cf5fc8-9qt8v\" (UID: \"42cba095-3e85-4cd0-a6fa-255c8086733b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9qt8v" Nov 03 20:53:56.159135 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:56.159105 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/42cba095-3e85-4cd0-a6fa-255c8086733b-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-9qt8v\" (UID: \"42cba095-3e85-4cd0-a6fa-255c8086733b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9qt8v" Nov 03 20:53:56.159135 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:56.159118 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42cba095-3e85-4cd0-a6fa-255c8086733b-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-9qt8v\" (UID: \"42cba095-3e85-4cd0-a6fa-255c8086733b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9qt8v" Nov 03 20:53:56.206240 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:56.206194 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 03 20:53:56.207192 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:56.207168 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-9qt8v"] Nov 03 20:53:56.209985 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:56.209963 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42cba095-3e85-4cd0-a6fa-255c8086733b-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-9qt8v\" (UID: \"42cba095-3e85-4cd0-a6fa-255c8086733b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9qt8v" Nov 03 20:53:56.217132 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:56.217108 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2xkr\" (UniqueName: \"kubernetes.io/projected/42cba095-3e85-4cd0-a6fa-255c8086733b-kube-api-access-d2xkr\") pod \"kserve-module-controller-manager-7566cf5fc8-9qt8v\" (UID: \"42cba095-3e85-4cd0-a6fa-255c8086733b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9qt8v" Nov 03 20:53:56.326854 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:56.326769 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9qt8v" Nov 03 20:53:56.738535 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:56.738510 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-9qt8v"] Nov 03 20:53:56.741180 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:53:56.741149 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42cba095_3e85_4cd0_a6fa_255c8086733b.slice/crio-09f8d81b9784fa5fb6a83c724a9d809c2f709baed96ecf764df2cfa277b464ac WatchSource:0}: Error finding container 09f8d81b9784fa5fb6a83c724a9d809c2f709baed96ecf764df2cfa277b464ac: Status 404 returned error can't find the container with id 09f8d81b9784fa5fb6a83c724a9d809c2f709baed96ecf764df2cfa277b464ac Nov 03 20:53:56.955765 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:56.955729 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9qt8v" event={"ID":"42cba095-3e85-4cd0-a6fa-255c8086733b","Type":"ContainerStarted","Data":"09f8d81b9784fa5fb6a83c724a9d809c2f709baed96ecf764df2cfa277b464ac"} Nov 03 20:53:58.966055 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:58.965962 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9qt8v" event={"ID":"42cba095-3e85-4cd0-a6fa-255c8086733b","Type":"ContainerStarted","Data":"d82cf9aa3fd89cd53e659fe40e0f91d49b73a101432c2d6019e24bc3aec61c48"} Nov 03 20:53:58.966532 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:58.966112 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9qt8v" Nov 03 20:53:59.128618 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:53:59.128561 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9qt8v" podStartSLOduration=2.206286403 podStartE2EDuration="4.12853842s" totalImagesPullingTime="1.922252017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:59.127484927 +0000 UTC m=+333.595263682" watchObservedRunningTime="2026-11-03 20:53:59.12853842 +0000 UTC m=+333.596317196" Nov 03 20:54:10.617876 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:54:10.617840 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 20:54:19.971047 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:19.971011 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9qt8v" Nov 03 20:54:40.617900 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:54:40.617816 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 20:54:57.317628 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:57.317582 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b"] Nov 03 20:54:57.323867 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:57.323654 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b" Nov 03 20:54:57.418879 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:57.418824 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbc7c3d8-118e-4425-9d20-1547e5a6078c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqd6b\" (UID: \"fbc7c3d8-118e-4425-9d20-1547e5a6078c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b" Nov 03 20:54:57.419107 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:57.418896 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbc7c3d8-118e-4425-9d20-1547e5a6078c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqd6b\" (UID: \"fbc7c3d8-118e-4425-9d20-1547e5a6078c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b" Nov 03 20:54:57.419107 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:57.418931 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fbc7c3d8-118e-4425-9d20-1547e5a6078c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqd6b\" (UID: \"fbc7c3d8-118e-4425-9d20-1547e5a6078c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b" Nov 03 20:54:57.419107 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:57.418967 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fbc7c3d8-118e-4425-9d20-1547e5a6078c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqd6b\" (UID: \"fbc7c3d8-118e-4425-9d20-1547e5a6078c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b" Nov 03 20:54:57.419107 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:57.419015 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fbc7c3d8-118e-4425-9d20-1547e5a6078c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqd6b\" (UID: \"fbc7c3d8-118e-4425-9d20-1547e5a6078c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b" Nov 03 20:54:57.419107 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:57.419107 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6hm4\" (UniqueName: \"kubernetes.io/projected/fbc7c3d8-118e-4425-9d20-1547e5a6078c-kube-api-access-v6hm4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqd6b\" (UID: \"fbc7c3d8-118e-4425-9d20-1547e5a6078c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b" Nov 03 20:54:57.419408 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:57.419142 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fbc7c3d8-118e-4425-9d20-1547e5a6078c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqd6b\" (UID: \"fbc7c3d8-118e-4425-9d20-1547e5a6078c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b" Nov 03 20:54:57.422235 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:57.422202 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b"] Nov 03 20:54:57.519921 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:57.519886 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbc7c3d8-118e-4425-9d20-1547e5a6078c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqd6b\" (UID: \"fbc7c3d8-118e-4425-9d20-1547e5a6078c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b" Nov 03 20:54:57.520135 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:57.519935 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbc7c3d8-118e-4425-9d20-1547e5a6078c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqd6b\" (UID: \"fbc7c3d8-118e-4425-9d20-1547e5a6078c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b" Nov 03 20:54:57.520135 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:57.519959 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fbc7c3d8-118e-4425-9d20-1547e5a6078c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqd6b\" (UID: \"fbc7c3d8-118e-4425-9d20-1547e5a6078c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b" Nov 03 20:54:57.520135 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:57.519983 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fbc7c3d8-118e-4425-9d20-1547e5a6078c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqd6b\" (UID: \"fbc7c3d8-118e-4425-9d20-1547e5a6078c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b" Nov 03 20:54:57.520135 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:57.520011 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fbc7c3d8-118e-4425-9d20-1547e5a6078c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqd6b\" (UID: \"fbc7c3d8-118e-4425-9d20-1547e5a6078c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b" Nov 03 20:54:57.520135 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:57.520062 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6hm4\" (UniqueName: \"kubernetes.io/projected/fbc7c3d8-118e-4425-9d20-1547e5a6078c-kube-api-access-v6hm4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqd6b\" (UID: \"fbc7c3d8-118e-4425-9d20-1547e5a6078c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b" Nov 03 20:54:57.520135 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:57.520123 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fbc7c3d8-118e-4425-9d20-1547e5a6078c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqd6b\" (UID: \"fbc7c3d8-118e-4425-9d20-1547e5a6078c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b" Nov 03 20:54:57.520864 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:57.520839 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fbc7c3d8-118e-4425-9d20-1547e5a6078c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqd6b\" (UID: \"fbc7c3d8-118e-4425-9d20-1547e5a6078c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b" Nov 03 20:54:57.522578 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:57.522551 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fbc7c3d8-118e-4425-9d20-1547e5a6078c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqd6b\" (UID: \"fbc7c3d8-118e-4425-9d20-1547e5a6078c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b" Nov 03 20:54:57.522671 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:57.522622 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fbc7c3d8-118e-4425-9d20-1547e5a6078c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqd6b\" (UID: \"fbc7c3d8-118e-4425-9d20-1547e5a6078c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b" Nov 03 20:54:57.522728 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:57.522680 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbc7c3d8-118e-4425-9d20-1547e5a6078c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqd6b\" (UID: \"fbc7c3d8-118e-4425-9d20-1547e5a6078c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b" Nov 03 20:54:57.522826 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:57.522805 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fbc7c3d8-118e-4425-9d20-1547e5a6078c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqd6b\" (UID: \"fbc7c3d8-118e-4425-9d20-1547e5a6078c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b" Nov 03 20:54:57.582427 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:57.582357 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbc7c3d8-118e-4425-9d20-1547e5a6078c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqd6b\" (UID: \"fbc7c3d8-118e-4425-9d20-1547e5a6078c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b" Nov 03 20:54:57.582627 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:57.582607 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6hm4\" (UniqueName: \"kubernetes.io/projected/fbc7c3d8-118e-4425-9d20-1547e5a6078c-kube-api-access-v6hm4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqd6b\" (UID: \"fbc7c3d8-118e-4425-9d20-1547e5a6078c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b" Nov 03 20:54:57.635519 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:57.635483 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b" Nov 03 20:54:58.050109 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:58.050036 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b"] Nov 03 20:54:58.052832 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:54:58.052804 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc7c3d8_118e_4425_9d20_1547e5a6078c.slice/crio-8ca3b5c5f503a777e36a917af402f89f2f992683737df4e21b88bd4547244fb1 WatchSource:0}: Error finding container 8ca3b5c5f503a777e36a917af402f89f2f992683737df4e21b88bd4547244fb1: Status 404 returned error can't find the container with id 8ca3b5c5f503a777e36a917af402f89f2f992683737df4e21b88bd4547244fb1 Nov 03 20:54:58.055038 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:58.055000 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:54:58.055140 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:58.055070 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:54:58.207913 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:58.207879 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b" event={"ID":"fbc7c3d8-118e-4425-9d20-1547e5a6078c","Type":"ContainerStarted","Data":"2518d94550dfd49bfca655f03d3ba09178bae1273df31c74800555386b14e147"} Nov 03 20:54:58.208121 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:58.207931 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b" event={"ID":"fbc7c3d8-118e-4425-9d20-1547e5a6078c","Type":"ContainerStarted","Data":"8ca3b5c5f503a777e36a917af402f89f2f992683737df4e21b88bd4547244fb1"} Nov 03 20:54:58.208121 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:58.208035 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b" Nov 03 20:54:58.331466 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:58.331371 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b" podStartSLOduration=1.331355205 podStartE2EDuration="1.331355205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:58.330344768 +0000 UTC m=+392.798123503" watchObservedRunningTime="2026-11-03 20:54:58.331355205 +0000 UTC m=+392.799133927" Nov 03 20:54:59.214960 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:59.214927 5615 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-lqd6b container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:54:59.215160 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:54:59.215017 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b" podUID="fbc7c3d8-118e-4425-9d20-1547e5a6078c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:55:02.214662 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:02.214629 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqd6b" Nov 03 20:55:02.426170 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:02.426124 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz"] Nov 03 20:55:02.426418 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:02.426370 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" podUID="4ffea0c2-79af-4d03-9948-6bc6f1223e6a" containerName="discovery" containerID="cri-o://d2b1e207290d200a63a37896d325a811391a9ffda2460f62c56c244f859e4a66" gracePeriod=30 Nov 03 20:55:02.678180 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:02.678147 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" Nov 03 20:55:02.771337 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:02.771230 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-local-certs\") pod \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\" (UID: \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\") " Nov 03 20:55:02.771337 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:02.771287 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-istio-kubeconfig\") pod \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\" (UID: \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\") " Nov 03 20:55:02.771337 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:02.771326 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-kube-api-access-bzf2m\" (UniqueName: \"kubernetes.io/projected/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-kube-api-access-bzf2m\") pod \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\" (UID: \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\") " Nov 03 20:55:02.771625 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:02.771386 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-istio-csr-ca-configmap\") pod \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\" (UID: \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\") " Nov 03 20:55:02.771625 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:02.771424 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-istio-token\" (UniqueName: \"kubernetes.io/projected/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-istio-token\") pod \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\" (UID: \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\") " Nov 03 20:55:02.771625 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:02.771482 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-istio-csr-dns-cert\") pod \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\" (UID: \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\") " Nov 03 20:55:02.771625 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:02.771567 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-cacerts\" (UniqueName: \"kubernetes.io/secret/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-cacerts\") pod \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\" (UID: \"4ffea0c2-79af-4d03-9948-6bc6f1223e6a\") " Nov 03 20:55:02.772119 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:02.771838 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-istio-csr-ca-configmap" pod "4ffea0c2-79af-4d03-9948-6bc6f1223e6a" (UID: "4ffea0c2-79af-4d03-9948-6bc6f1223e6a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:02.772119 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:02.771926 5615 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-istio-csr-ca-configmap\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:02.774098 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:02.774043 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-local-certs" pod "4ffea0c2-79af-4d03-9948-6bc6f1223e6a" (UID: "4ffea0c2-79af-4d03-9948-6bc6f1223e6a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:02.774204 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:02.774093 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-istio-kubeconfig" pod "4ffea0c2-79af-4d03-9948-6bc6f1223e6a" (UID: "4ffea0c2-79af-4d03-9948-6bc6f1223e6a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:55:02.774204 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:02.774052 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-kube-api-access-bzf2m" pod "4ffea0c2-79af-4d03-9948-6bc6f1223e6a" (UID: "4ffea0c2-79af-4d03-9948-6bc6f1223e6a"). InnerVolumeSpecName "kube-api-access-bzf2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:02.774204 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:02.774121 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-istio-token" pod "4ffea0c2-79af-4d03-9948-6bc6f1223e6a" (UID: "4ffea0c2-79af-4d03-9948-6bc6f1223e6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:02.774204 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:02.774124 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-istio-csr-dns-cert" pod "4ffea0c2-79af-4d03-9948-6bc6f1223e6a" (UID: "4ffea0c2-79af-4d03-9948-6bc6f1223e6a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:55:02.774372 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:02.774357 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-cacerts" pod "4ffea0c2-79af-4d03-9948-6bc6f1223e6a" (UID: "4ffea0c2-79af-4d03-9948-6bc6f1223e6a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:55:02.872542 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:02.872506 5615 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-istio-csr-dns-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:02.872542 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:02.872539 5615 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-cacerts\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:02.872542 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:02.872548 5615 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-local-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:02.872785 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:02.872559 5615 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-istio-kubeconfig\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:02.872785 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:02.872569 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzf2m\" (UniqueName: \"kubernetes.io/projected/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-kube-api-access-bzf2m\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:02.872785 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:02.872579 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ffea0c2-79af-4d03-9948-6bc6f1223e6a-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:03.225892 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.225811 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-x87fj"] Nov 03 20:55:03.226282 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.226194 5615 cpu_manager.go:414] "containerMap: removing container" podUID="4ffea0c2-79af-4d03-9948-6bc6f1223e6a" containerName="discovery" Nov 03 20:55:03.226282 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.226207 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ffea0c2-79af-4d03-9948-6bc6f1223e6a" containerName="discovery" Nov 03 20:55:03.226282 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.226269 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ffea0c2-79af-4d03-9948-6bc6f1223e6a" containerName="discovery" Nov 03 20:55:03.229728 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.229698 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-x87fj" Nov 03 20:55:03.232352 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.232286 5615 generic.go:356] "Generic (PLEG): container finished" podID="4ffea0c2-79af-4d03-9948-6bc6f1223e6a" containerID="d2b1e207290d200a63a37896d325a811391a9ffda2460f62c56c244f859e4a66" exitCode=0 Nov 03 20:55:03.232352 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.232344 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" Nov 03 20:55:03.232582 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.232435 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" event={"ID":"4ffea0c2-79af-4d03-9948-6bc6f1223e6a","Type":"ContainerDied","Data":"d2b1e207290d200a63a37896d325a811391a9ffda2460f62c56c244f859e4a66"} Nov 03 20:55:03.232582 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.232475 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz" event={"ID":"4ffea0c2-79af-4d03-9948-6bc6f1223e6a","Type":"ContainerDied","Data":"1cfda76d90b597a0c1ec226cf280c249477e58a74286a9ff97f65f6250da4f82"} Nov 03 20:55:03.232582 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.232499 5615 scope.go:122] "RemoveContainer" containerID="d2b1e207290d200a63a37896d325a811391a9ffda2460f62c56c244f859e4a66" Nov 03 20:55:03.249122 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.249096 5615 scope.go:122] "RemoveContainer" containerID="d2b1e207290d200a63a37896d325a811391a9ffda2460f62c56c244f859e4a66" Nov 03 20:55:03.249456 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:55:03.249434 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2b1e207290d200a63a37896d325a811391a9ffda2460f62c56c244f859e4a66\": container with ID starting with d2b1e207290d200a63a37896d325a811391a9ffda2460f62c56c244f859e4a66 not found: ID does not exist" containerID="d2b1e207290d200a63a37896d325a811391a9ffda2460f62c56c244f859e4a66" Nov 03 20:55:03.249542 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.249468 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2b1e207290d200a63a37896d325a811391a9ffda2460f62c56c244f859e4a66"} err="failed to get container status \"d2b1e207290d200a63a37896d325a811391a9ffda2460f62c56c244f859e4a66\": rpc error: code = NotFound desc = could not find container \"d2b1e207290d200a63a37896d325a811391a9ffda2460f62c56c244f859e4a66\": container with ID starting with d2b1e207290d200a63a37896d325a811391a9ffda2460f62c56c244f859e4a66 not found: ID does not exist" Nov 03 20:55:03.279324 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.279287 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-6lnkq\"" Nov 03 20:55:03.279479 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.279430 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 03 20:55:03.283847 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.283816 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-x87fj"] Nov 03 20:55:03.377702 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.377666 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79c0375b-54c7-4629-83da-f41b5b7e8da9-cert\") pod \"kserve-controller-manager-776cc66d47-x87fj\" (UID: \"79c0375b-54c7-4629-83da-f41b5b7e8da9\") " pod="opendatahub/kserve-controller-manager-776cc66d47-x87fj" Nov 03 20:55:03.377882 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.377707 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwg2l\" (UniqueName: \"kubernetes.io/projected/79c0375b-54c7-4629-83da-f41b5b7e8da9-kube-api-access-lwg2l\") pod \"kserve-controller-manager-776cc66d47-x87fj\" (UID: \"79c0375b-54c7-4629-83da-f41b5b7e8da9\") " pod="opendatahub/kserve-controller-manager-776cc66d47-x87fj" Nov 03 20:55:03.445782 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.445750 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz"] Nov 03 20:55:03.454159 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.454125 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltmzz"] Nov 03 20:55:03.459636 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.459596 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-lksbc"] Nov 03 20:55:03.463062 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.463044 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-lksbc" Nov 03 20:55:03.478862 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.478812 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79c0375b-54c7-4629-83da-f41b5b7e8da9-cert\") pod \"kserve-controller-manager-776cc66d47-x87fj\" (UID: \"79c0375b-54c7-4629-83da-f41b5b7e8da9\") " pod="opendatahub/kserve-controller-manager-776cc66d47-x87fj" Nov 03 20:55:03.478862 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.478839 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwg2l\" (UniqueName: \"kubernetes.io/projected/79c0375b-54c7-4629-83da-f41b5b7e8da9-kube-api-access-lwg2l\") pod \"kserve-controller-manager-776cc66d47-x87fj\" (UID: \"79c0375b-54c7-4629-83da-f41b5b7e8da9\") " pod="opendatahub/kserve-controller-manager-776cc66d47-x87fj" Nov 03 20:55:03.481147 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.481123 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79c0375b-54c7-4629-83da-f41b5b7e8da9-cert\") pod \"kserve-controller-manager-776cc66d47-x87fj\" (UID: \"79c0375b-54c7-4629-83da-f41b5b7e8da9\") " pod="opendatahub/kserve-controller-manager-776cc66d47-x87fj" Nov 03 20:55:03.512393 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.512342 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 03 20:55:03.512546 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.512400 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-kkxdd\"" Nov 03 20:55:03.517759 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.517733 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-lksbc"] Nov 03 20:55:03.536973 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.536942 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwg2l\" (UniqueName: \"kubernetes.io/projected/79c0375b-54c7-4629-83da-f41b5b7e8da9-kube-api-access-lwg2l\") pod \"kserve-controller-manager-776cc66d47-x87fj\" (UID: \"79c0375b-54c7-4629-83da-f41b5b7e8da9\") " pod="opendatahub/kserve-controller-manager-776cc66d47-x87fj" Nov 03 20:55:03.553275 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.553238 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-x87fj" Nov 03 20:55:03.580392 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.580345 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddbt9\" (UniqueName: \"kubernetes.io/projected/db528ef3-1b85-4e45-be0c-602a4eeb95c6-kube-api-access-ddbt9\") pod \"model-serving-api-75bd89fdfc-lksbc\" (UID: \"db528ef3-1b85-4e45-be0c-602a4eeb95c6\") " pod="opendatahub/model-serving-api-75bd89fdfc-lksbc" Nov 03 20:55:03.580531 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.580472 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db528ef3-1b85-4e45-be0c-602a4eeb95c6-tls-certs\") pod \"model-serving-api-75bd89fdfc-lksbc\" (UID: \"db528ef3-1b85-4e45-be0c-602a4eeb95c6\") " pod="opendatahub/model-serving-api-75bd89fdfc-lksbc" Nov 03 20:55:03.650203 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.650167 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ffea0c2-79af-4d03-9948-6bc6f1223e6a" path="/var/lib/kubelet/pods/4ffea0c2-79af-4d03-9948-6bc6f1223e6a/volumes" Nov 03 20:55:03.681845 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.681808 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddbt9\" (UniqueName: \"kubernetes.io/projected/db528ef3-1b85-4e45-be0c-602a4eeb95c6-kube-api-access-ddbt9\") pod \"model-serving-api-75bd89fdfc-lksbc\" (UID: \"db528ef3-1b85-4e45-be0c-602a4eeb95c6\") " pod="opendatahub/model-serving-api-75bd89fdfc-lksbc" Nov 03 20:55:03.682009 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.681857 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db528ef3-1b85-4e45-be0c-602a4eeb95c6-tls-certs\") pod \"model-serving-api-75bd89fdfc-lksbc\" (UID: \"db528ef3-1b85-4e45-be0c-602a4eeb95c6\") " pod="opendatahub/model-serving-api-75bd89fdfc-lksbc" Nov 03 20:55:03.684285 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.684264 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db528ef3-1b85-4e45-be0c-602a4eeb95c6-tls-certs\") pod \"model-serving-api-75bd89fdfc-lksbc\" (UID: \"db528ef3-1b85-4e45-be0c-602a4eeb95c6\") " pod="opendatahub/model-serving-api-75bd89fdfc-lksbc" Nov 03 20:55:03.740018 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.739991 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddbt9\" (UniqueName: \"kubernetes.io/projected/db528ef3-1b85-4e45-be0c-602a4eeb95c6-kube-api-access-ddbt9\") pod \"model-serving-api-75bd89fdfc-lksbc\" (UID: \"db528ef3-1b85-4e45-be0c-602a4eeb95c6\") " pod="opendatahub/model-serving-api-75bd89fdfc-lksbc" Nov 03 20:55:03.775899 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.775865 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-lksbc" Nov 03 20:55:03.917756 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.917730 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-x87fj"] Nov 03 20:55:03.920123 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:55:03.920093 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c0375b_54c7_4629_83da_f41b5b7e8da9.slice/crio-a4c4920d2f20729db0375e3d2968e745a46ecd3ec2d1e1c8c2d00e4b9317bef8 WatchSource:0}: Error finding container a4c4920d2f20729db0375e3d2968e745a46ecd3ec2d1e1c8c2d00e4b9317bef8: Status 404 returned error can't find the container with id a4c4920d2f20729db0375e3d2968e745a46ecd3ec2d1e1c8c2d00e4b9317bef8 Nov 03 20:55:03.921518 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:03.921497 5615 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:55:04.140923 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:04.140802 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-lksbc"] Nov 03 20:55:04.143576 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:55:04.143549 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb528ef3_1b85_4e45_be0c_602a4eeb95c6.slice/crio-b17b85ae8d25173bafbbfb83e6ee979bd859c8ebb67e276ad75ffcabf0608359 WatchSource:0}: Error finding container b17b85ae8d25173bafbbfb83e6ee979bd859c8ebb67e276ad75ffcabf0608359: Status 404 returned error can't find the container with id b17b85ae8d25173bafbbfb83e6ee979bd859c8ebb67e276ad75ffcabf0608359 Nov 03 20:55:04.238019 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:04.237976 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-lksbc" event={"ID":"db528ef3-1b85-4e45-be0c-602a4eeb95c6","Type":"ContainerStarted","Data":"b17b85ae8d25173bafbbfb83e6ee979bd859c8ebb67e276ad75ffcabf0608359"} Nov 03 20:55:04.239980 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:04.239952 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-x87fj" event={"ID":"79c0375b-54c7-4629-83da-f41b5b7e8da9","Type":"ContainerStarted","Data":"a4c4920d2f20729db0375e3d2968e745a46ecd3ec2d1e1c8c2d00e4b9317bef8"} Nov 03 20:55:07.256803 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:07.256770 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-lksbc" event={"ID":"db528ef3-1b85-4e45-be0c-602a4eeb95c6","Type":"ContainerStarted","Data":"1a094eb595398b04e6931eb3665817dac1d8e4baf7f6419329b8dc5980283e88"} Nov 03 20:55:07.257298 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:07.256863 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-lksbc" Nov 03 20:55:07.258233 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:07.258208 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-x87fj" event={"ID":"79c0375b-54c7-4629-83da-f41b5b7e8da9","Type":"ContainerStarted","Data":"ee52bfb4b10d7ac790f7dc87d3bccdea145935db92c21dfe942c2b9600a741b7"} Nov 03 20:55:07.258346 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:07.258331 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-x87fj" Nov 03 20:55:07.368018 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:07.367962 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-lksbc" podStartSLOduration=2.051673924 podStartE2EDuration="4.367948354s" totalImagesPullingTime="2.31627443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:07.367419113 +0000 UTC m=+401.835197837" watchObservedRunningTime="2026-11-03 20:55:07.367948354 +0000 UTC m=+401.835727055" Nov 03 20:55:07.478282 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:07.478224 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-x87fj" podStartSLOduration=1.982796176 podStartE2EDuration="4.478210287s" totalImagesPullingTime="2.495414111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:07.47762207 +0000 UTC m=+401.945400805" watchObservedRunningTime="2026-11-03 20:55:07.478210287 +0000 UTC m=+401.945988989" Nov 03 20:55:10.618351 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:55:10.618314 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 20:55:18.268143 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:18.268114 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-lksbc" Nov 03 20:55:31.277423 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:31.277391 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7c97fc4d98-hm842"] Nov 03 20:55:31.287742 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:31.287707 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-hm842" Nov 03 20:55:31.336893 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:31.336854 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c97fc4d98-hm842"] Nov 03 20:55:31.382834 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:31.382793 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:55:31.382834 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:31.382807 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:55:31.384388 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:31.384371 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6whcf\"" Nov 03 20:55:31.424434 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:31.424402 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 20:55:31.424434 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:31.424402 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:55:31.430824 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:31.430795 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/789f79fa-622f-445e-9851-994ffd94dc9b-maas-api-tls\") pod \"maas-api-7c97fc4d98-hm842\" (UID: \"789f79fa-622f-445e-9851-994ffd94dc9b\") " pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-hm842" Nov 03 20:55:31.430961 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:31.430838 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/789f79fa-622f-445e-9851-994ffd94dc9b-openshift-service-ca\") pod \"maas-api-7c97fc4d98-hm842\" (UID: \"789f79fa-622f-445e-9851-994ffd94dc9b\") " pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-hm842" Nov 03 20:55:31.430961 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:31.430919 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkfjv\" (UniqueName: \"kubernetes.io/projected/789f79fa-622f-445e-9851-994ffd94dc9b-kube-api-access-pkfjv\") pod \"maas-api-7c97fc4d98-hm842\" (UID: \"789f79fa-622f-445e-9851-994ffd94dc9b\") " pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-hm842" Nov 03 20:55:31.431041 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:31.430982 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/789f79fa-622f-445e-9851-994ffd94dc9b-metrics-certs\") pod \"maas-api-7c97fc4d98-hm842\" (UID: \"789f79fa-622f-445e-9851-994ffd94dc9b\") " pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-hm842" Nov 03 20:55:31.531916 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:31.531826 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/789f79fa-622f-445e-9851-994ffd94dc9b-maas-api-tls\") pod \"maas-api-7c97fc4d98-hm842\" (UID: \"789f79fa-622f-445e-9851-994ffd94dc9b\") " pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-hm842" Nov 03 20:55:31.531916 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:31.531881 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/789f79fa-622f-445e-9851-994ffd94dc9b-openshift-service-ca\") pod \"maas-api-7c97fc4d98-hm842\" (UID: \"789f79fa-622f-445e-9851-994ffd94dc9b\") " pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-hm842" Nov 03 20:55:31.532169 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:31.531922 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkfjv\" (UniqueName: \"kubernetes.io/projected/789f79fa-622f-445e-9851-994ffd94dc9b-kube-api-access-pkfjv\") pod \"maas-api-7c97fc4d98-hm842\" (UID: \"789f79fa-622f-445e-9851-994ffd94dc9b\") " pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-hm842" Nov 03 20:55:31.532169 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:31.531942 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/789f79fa-622f-445e-9851-994ffd94dc9b-metrics-certs\") pod \"maas-api-7c97fc4d98-hm842\" (UID: \"789f79fa-622f-445e-9851-994ffd94dc9b\") " pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-hm842" Nov 03 20:55:31.532666 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:31.532642 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/789f79fa-622f-445e-9851-994ffd94dc9b-openshift-service-ca\") pod \"maas-api-7c97fc4d98-hm842\" (UID: \"789f79fa-622f-445e-9851-994ffd94dc9b\") " pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-hm842" Nov 03 20:55:31.534410 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:31.534388 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/789f79fa-622f-445e-9851-994ffd94dc9b-metrics-certs\") pod \"maas-api-7c97fc4d98-hm842\" (UID: \"789f79fa-622f-445e-9851-994ffd94dc9b\") " pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-hm842" Nov 03 20:55:31.534410 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:31.534405 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/789f79fa-622f-445e-9851-994ffd94dc9b-maas-api-tls\") pod \"maas-api-7c97fc4d98-hm842\" (UID: \"789f79fa-622f-445e-9851-994ffd94dc9b\") " pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-hm842" Nov 03 20:55:31.590133 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:31.590094 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkfjv\" (UniqueName: \"kubernetes.io/projected/789f79fa-622f-445e-9851-994ffd94dc9b-kube-api-access-pkfjv\") pod \"maas-api-7c97fc4d98-hm842\" (UID: \"789f79fa-622f-445e-9851-994ffd94dc9b\") " pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-hm842" Nov 03 20:55:31.601146 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:31.601117 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-hm842" Nov 03 20:55:31.814560 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:31.814481 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5696f576-mwgpm"] Nov 03 20:55:31.818704 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:31.818679 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5696f576-mwgpm" Nov 03 20:55:31.869919 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:31.869761 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 20:55:31.871170 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:31.871141 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5696f576-mwgpm"] Nov 03 20:55:31.935688 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:31.935648 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml72g\" (UniqueName: \"kubernetes.io/projected/8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3-kube-api-access-ml72g\") pod \"authorino-5696f576-mwgpm\" (UID: \"8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3\") " pod="kuadrant-system/authorino-5696f576-mwgpm" Nov 03 20:55:31.935864 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:31.935756 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3-tls-cert\") pod \"authorino-5696f576-mwgpm\" (UID: \"8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3\") " pod="kuadrant-system/authorino-5696f576-mwgpm" Nov 03 20:55:31.986809 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:31.986737 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c97fc4d98-hm842"] Nov 03 20:55:31.989595 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:55:31.989569 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789f79fa_622f_445e_9851_994ffd94dc9b.slice/crio-51f4d00df55beef5c8dbeafe13a69fb760d1e52937ad4358dfadb2f4c4b69d07 WatchSource:0}: Error finding container 51f4d00df55beef5c8dbeafe13a69fb760d1e52937ad4358dfadb2f4c4b69d07: Status 404 returned error can't find the container with id 51f4d00df55beef5c8dbeafe13a69fb760d1e52937ad4358dfadb2f4c4b69d07 Nov 03 20:55:32.037094 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:32.037042 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml72g\" (UniqueName: \"kubernetes.io/projected/8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3-kube-api-access-ml72g\") pod \"authorino-5696f576-mwgpm\" (UID: \"8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3\") " pod="kuadrant-system/authorino-5696f576-mwgpm" Nov 03 20:55:32.037254 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:32.037125 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3-tls-cert\") pod \"authorino-5696f576-mwgpm\" (UID: \"8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3\") " pod="kuadrant-system/authorino-5696f576-mwgpm" Nov 03 20:55:32.039563 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:32.039533 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3-tls-cert\") pod \"authorino-5696f576-mwgpm\" (UID: \"8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3\") " pod="kuadrant-system/authorino-5696f576-mwgpm" Nov 03 20:55:32.095758 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:32.095681 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml72g\" (UniqueName: \"kubernetes.io/projected/8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3-kube-api-access-ml72g\") pod \"authorino-5696f576-mwgpm\" (UID: \"8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3\") " pod="kuadrant-system/authorino-5696f576-mwgpm" Nov 03 20:55:32.131333 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:32.131308 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5696f576-mwgpm" Nov 03 20:55:32.370178 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:32.370098 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-hm842" event={"ID":"789f79fa-622f-445e-9851-994ffd94dc9b","Type":"ContainerStarted","Data":"51f4d00df55beef5c8dbeafe13a69fb760d1e52937ad4358dfadb2f4c4b69d07"} Nov 03 20:55:32.492995 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:32.492887 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5696f576-mwgpm"] Nov 03 20:55:32.495601 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:55:32.495570 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d9784e5_3fcf_40d4_9416_cdcf6cf1acb3.slice/crio-01edb3254cae24d4d514699de686647e1dfc8251043fa0bb47bf1c9a42dda0e5 WatchSource:0}: Error finding container 01edb3254cae24d4d514699de686647e1dfc8251043fa0bb47bf1c9a42dda0e5: Status 404 returned error can't find the container with id 01edb3254cae24d4d514699de686647e1dfc8251043fa0bb47bf1c9a42dda0e5 Nov 03 20:55:33.375613 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:33.375575 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5696f576-mwgpm" event={"ID":"8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3","Type":"ContainerStarted","Data":"01edb3254cae24d4d514699de686647e1dfc8251043fa0bb47bf1c9a42dda0e5"} Nov 03 20:55:34.383476 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:34.383434 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5696f576-mwgpm" event={"ID":"8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3","Type":"ContainerStarted","Data":"ee93e11ed029b37f8297e7972989f1014ece5c7af01dec7768ba8f58761e14d7"} Nov 03 20:55:34.493226 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:34.493172 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5696f576-mwgpm" podStartSLOduration=1.778424944 podStartE2EDuration="3.493155324s" totalImagesPullingTime="1.71473038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:34.49303946 +0000 UTC m=+428.960818184" watchObservedRunningTime="2026-11-03 20:55:34.493155324 +0000 UTC m=+428.960934047" Nov 03 20:55:34.520020 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:34.519448 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-snqk8"] Nov 03 20:55:34.520020 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:34.519727 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-snqk8" podUID="32026c4b-92f7-493a-8067-cabb159f7eed" containerName="authorino" containerID="cri-o://9100a42a2fa93a4d086523a20bab0d350db337979d54d6771143342909dc07c4" gracePeriod=30 Nov 03 20:55:34.788162 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:34.788111 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-snqk8" Nov 03 20:55:34.865666 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:34.865623 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32026c4b-92f7-493a-8067-cabb159f7eed-kube-api-access-4t2fc\" (UniqueName: \"kubernetes.io/projected/32026c4b-92f7-493a-8067-cabb159f7eed-kube-api-access-4t2fc\") pod \"32026c4b-92f7-493a-8067-cabb159f7eed\" (UID: \"32026c4b-92f7-493a-8067-cabb159f7eed\") " Nov 03 20:55:34.868375 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:34.868343 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32026c4b-92f7-493a-8067-cabb159f7eed-kube-api-access-4t2fc" pod "32026c4b-92f7-493a-8067-cabb159f7eed" (UID: "32026c4b-92f7-493a-8067-cabb159f7eed"). InnerVolumeSpecName "kube-api-access-4t2fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:34.967348 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:34.967304 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t2fc\" (UniqueName: \"kubernetes.io/projected/32026c4b-92f7-493a-8067-cabb159f7eed-kube-api-access-4t2fc\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:35.389381 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:35.389344 5615 generic.go:356] "Generic (PLEG): container finished" podID="32026c4b-92f7-493a-8067-cabb159f7eed" containerID="9100a42a2fa93a4d086523a20bab0d350db337979d54d6771143342909dc07c4" exitCode=0 Nov 03 20:55:35.389806 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:35.389405 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-snqk8" Nov 03 20:55:35.389806 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:35.389439 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-snqk8" event={"ID":"32026c4b-92f7-493a-8067-cabb159f7eed","Type":"ContainerDied","Data":"9100a42a2fa93a4d086523a20bab0d350db337979d54d6771143342909dc07c4"} Nov 03 20:55:35.389806 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:35.389496 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-snqk8" event={"ID":"32026c4b-92f7-493a-8067-cabb159f7eed","Type":"ContainerDied","Data":"8342f8620655108cbe04b97aad2d5179c2354675cb3f1789702a8493541802c1"} Nov 03 20:55:35.389806 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:35.389521 5615 scope.go:122] "RemoveContainer" containerID="9100a42a2fa93a4d086523a20bab0d350db337979d54d6771143342909dc07c4" Nov 03 20:55:35.398706 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:35.398682 5615 scope.go:122] "RemoveContainer" containerID="9100a42a2fa93a4d086523a20bab0d350db337979d54d6771143342909dc07c4" Nov 03 20:55:35.398977 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:55:35.398958 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9100a42a2fa93a4d086523a20bab0d350db337979d54d6771143342909dc07c4\": container with ID starting with 9100a42a2fa93a4d086523a20bab0d350db337979d54d6771143342909dc07c4 not found: ID does not exist" containerID="9100a42a2fa93a4d086523a20bab0d350db337979d54d6771143342909dc07c4" Nov 03 20:55:35.399019 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:35.398987 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9100a42a2fa93a4d086523a20bab0d350db337979d54d6771143342909dc07c4"} err="failed to get container status \"9100a42a2fa93a4d086523a20bab0d350db337979d54d6771143342909dc07c4\": rpc error: code = NotFound desc = could not find container \"9100a42a2fa93a4d086523a20bab0d350db337979d54d6771143342909dc07c4\": container with ID starting with 9100a42a2fa93a4d086523a20bab0d350db337979d54d6771143342909dc07c4 not found: ID does not exist" Nov 03 20:55:35.597437 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:35.597403 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-snqk8"] Nov 03 20:55:35.598198 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:35.598175 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-snqk8"] Nov 03 20:55:35.650884 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:35.650850 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32026c4b-92f7-493a-8067-cabb159f7eed" path="/var/lib/kubelet/pods/32026c4b-92f7-493a-8067-cabb159f7eed/volumes" Nov 03 20:55:38.268685 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:38.268658 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-x87fj" Nov 03 20:55:38.406910 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:38.406870 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-hm842" event={"ID":"789f79fa-622f-445e-9851-994ffd94dc9b","Type":"ContainerStarted","Data":"0ac92368d2f79bfdb36788b906be4dd9049de9853817d071fb531eea7aa8cd41"} Nov 03 20:55:38.407145 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:38.406951 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-hm842" Nov 03 20:55:38.519377 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:38.519255 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-hm842" podStartSLOduration=1.904739768 podStartE2EDuration="7.519239418s" totalImagesPullingTime="5.61449965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:38.518547031 +0000 UTC m=+432.986325754" watchObservedRunningTime="2026-11-03 20:55:38.519239418 +0000 UTC m=+432.987018141" Nov 03 20:55:40.617458 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:55:40.617419 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 20:55:44.417215 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:55:44.417180 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-hm842" Nov 03 20:56:10.617749 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:56:10.617666 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 20:56:40.617866 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:56:40.617826 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 20:56:51.982825 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:56:51.982786 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-chtmx"] Nov 03 20:56:51.983282 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:56:51.983177 5615 cpu_manager.go:414] "containerMap: removing container" podUID="32026c4b-92f7-493a-8067-cabb159f7eed" containerName="authorino" Nov 03 20:56:51.983282 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:56:51.983189 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32026c4b-92f7-493a-8067-cabb159f7eed" containerName="authorino" Nov 03 20:56:51.983282 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:56:51.983274 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="32026c4b-92f7-493a-8067-cabb159f7eed" containerName="authorino" Nov 03 20:56:51.986233 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:56:51.986214 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-chtmx" Nov 03 20:56:52.036169 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:56:52.036137 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 03 20:56:52.083200 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:56:52.083167 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 03 20:56:52.083388 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:56:52.083252 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-tphjn\"" Nov 03 20:56:52.084376 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:56:52.084352 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-chtmx"] Nov 03 20:56:52.161094 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:56:52.161043 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fdde323-ed8a-431b-915c-7a5a705903a0-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-chtmx\" (UID: \"3fdde323-ed8a-431b-915c-7a5a705903a0\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-chtmx" Nov 03 20:56:52.161262 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:56:52.161171 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zqgj\" (UniqueName: \"kubernetes.io/projected/3fdde323-ed8a-431b-915c-7a5a705903a0-kube-api-access-5zqgj\") pod \"ai-gateway-controller-5d8dddb676-chtmx\" (UID: \"3fdde323-ed8a-431b-915c-7a5a705903a0\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-chtmx" Nov 03 20:56:52.262134 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:56:52.262020 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fdde323-ed8a-431b-915c-7a5a705903a0-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-chtmx\" (UID: \"3fdde323-ed8a-431b-915c-7a5a705903a0\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-chtmx" Nov 03 20:56:52.262134 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:56:52.262064 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zqgj\" (UniqueName: \"kubernetes.io/projected/3fdde323-ed8a-431b-915c-7a5a705903a0-kube-api-access-5zqgj\") pod \"ai-gateway-controller-5d8dddb676-chtmx\" (UID: \"3fdde323-ed8a-431b-915c-7a5a705903a0\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-chtmx" Nov 03 20:56:52.264546 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:56:52.264521 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fdde323-ed8a-431b-915c-7a5a705903a0-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-chtmx\" (UID: \"3fdde323-ed8a-431b-915c-7a5a705903a0\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-chtmx" Nov 03 20:56:52.321155 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:56:52.321116 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zqgj\" (UniqueName: \"kubernetes.io/projected/3fdde323-ed8a-431b-915c-7a5a705903a0-kube-api-access-5zqgj\") pod \"ai-gateway-controller-5d8dddb676-chtmx\" (UID: \"3fdde323-ed8a-431b-915c-7a5a705903a0\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-chtmx" Nov 03 20:56:52.599129 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:56:52.599012 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-chtmx" Nov 03 20:56:52.963627 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:56:52.963573 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-chtmx"] Nov 03 20:56:52.966012 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:56:52.965985 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fdde323_ed8a_431b_915c_7a5a705903a0.slice/crio-a99705a9ffc643af6870e5f782086bd4ec13e8f211c76bba66681431608cc9b7 WatchSource:0}: Error finding container a99705a9ffc643af6870e5f782086bd4ec13e8f211c76bba66681431608cc9b7: Status 404 returned error can't find the container with id a99705a9ffc643af6870e5f782086bd4ec13e8f211c76bba66681431608cc9b7 Nov 03 20:56:53.731348 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:56:53.731309 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-chtmx" event={"ID":"3fdde323-ed8a-431b-915c-7a5a705903a0","Type":"ContainerStarted","Data":"a99705a9ffc643af6870e5f782086bd4ec13e8f211c76bba66681431608cc9b7"} Nov 03 20:56:58.754263 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:56:58.754219 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-chtmx" event={"ID":"3fdde323-ed8a-431b-915c-7a5a705903a0","Type":"ContainerStarted","Data":"f38ddd017e8c4fc9bd162285bba5d25ced8493aed40417d9dd93d9ff6166629e"} Nov 03 20:56:58.754701 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:56:58.754364 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-chtmx" Nov 03 20:56:58.865500 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:56:58.865441 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d8dddb676-chtmx" podStartSLOduration=2.835780464 podStartE2EDuration="7.86542715s" totalImagesPullingTime="5.029646686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:58.864603387 +0000 UTC m=+513.332382112" watchObservedRunningTime="2026-11-03 20:56:58.86542715 +0000 UTC m=+513.333205873" Nov 03 20:57:01.179597 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:01.179559 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-jjmps"] Nov 03 20:57:01.183254 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:01.183232 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jjmps" Nov 03 20:57:01.237291 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:01.237261 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-jjmps"] Nov 03 20:57:01.246629 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:01.246594 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f60591a-fc5a-47d9-9899-93963284b5d6-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-jjmps\" (UID: \"2f60591a-fc5a-47d9-9899-93963284b5d6\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-jjmps" Nov 03 20:57:01.246775 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:01.246646 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkbml\" (UniqueName: \"kubernetes.io/projected/2f60591a-fc5a-47d9-9899-93963284b5d6-kube-api-access-vkbml\") pod \"payload-pre-processing-7b98bb678c-jjmps\" (UID: \"2f60591a-fc5a-47d9-9899-93963284b5d6\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-jjmps" Nov 03 20:57:01.281050 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:01.281019 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-kbdwc\"" Nov 03 20:57:01.281050 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:01.281019 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 20:57:01.329241 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:01.329207 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-r5zp7"] Nov 03 20:57:01.332787 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:01.332764 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-r5zp7" Nov 03 20:57:01.347833 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:01.347805 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f60591a-fc5a-47d9-9899-93963284b5d6-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-jjmps\" (UID: \"2f60591a-fc5a-47d9-9899-93963284b5d6\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-jjmps" Nov 03 20:57:01.347968 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:01.347856 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j28sv\" (UniqueName: \"kubernetes.io/projected/40e8212c-49a4-4a50-add6-0ee8a55429b2-kube-api-access-j28sv\") pod \"payload-processing-7bb9489676-r5zp7\" (UID: \"40e8212c-49a4-4a50-add6-0ee8a55429b2\") " pod="openshift-ingress/payload-processing-7bb9489676-r5zp7" Nov 03 20:57:01.347968 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:01.347894 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkbml\" (UniqueName: \"kubernetes.io/projected/2f60591a-fc5a-47d9-9899-93963284b5d6-kube-api-access-vkbml\") pod \"payload-pre-processing-7b98bb678c-jjmps\" (UID: \"2f60591a-fc5a-47d9-9899-93963284b5d6\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-jjmps" Nov 03 20:57:01.347968 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:01.347936 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e8212c-49a4-4a50-add6-0ee8a55429b2-plugins-config-volume\") pod \"payload-processing-7bb9489676-r5zp7\" (UID: \"40e8212c-49a4-4a50-add6-0ee8a55429b2\") " pod="openshift-ingress/payload-processing-7bb9489676-r5zp7" Nov 03 20:57:01.348554 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:01.348534 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f60591a-fc5a-47d9-9899-93963284b5d6-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-jjmps\" (UID: \"2f60591a-fc5a-47d9-9899-93963284b5d6\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-jjmps" Nov 03 20:57:01.384989 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:01.384957 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-r5zp7"] Nov 03 20:57:01.408309 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:01.408281 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkbml\" (UniqueName: \"kubernetes.io/projected/2f60591a-fc5a-47d9-9899-93963284b5d6-kube-api-access-vkbml\") pod \"payload-pre-processing-7b98bb678c-jjmps\" (UID: \"2f60591a-fc5a-47d9-9899-93963284b5d6\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-jjmps" Nov 03 20:57:01.448843 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:01.448757 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j28sv\" (UniqueName: \"kubernetes.io/projected/40e8212c-49a4-4a50-add6-0ee8a55429b2-kube-api-access-j28sv\") pod \"payload-processing-7bb9489676-r5zp7\" (UID: \"40e8212c-49a4-4a50-add6-0ee8a55429b2\") " pod="openshift-ingress/payload-processing-7bb9489676-r5zp7" Nov 03 20:57:01.448843 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:01.448807 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e8212c-49a4-4a50-add6-0ee8a55429b2-plugins-config-volume\") pod \"payload-processing-7bb9489676-r5zp7\" (UID: \"40e8212c-49a4-4a50-add6-0ee8a55429b2\") " pod="openshift-ingress/payload-processing-7bb9489676-r5zp7" Nov 03 20:57:01.449546 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:01.449519 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e8212c-49a4-4a50-add6-0ee8a55429b2-plugins-config-volume\") pod \"payload-processing-7bb9489676-r5zp7\" (UID: \"40e8212c-49a4-4a50-add6-0ee8a55429b2\") " pod="openshift-ingress/payload-processing-7bb9489676-r5zp7" Nov 03 20:57:01.495774 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:01.495733 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jjmps" Nov 03 20:57:01.507233 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:01.507138 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j28sv\" (UniqueName: \"kubernetes.io/projected/40e8212c-49a4-4a50-add6-0ee8a55429b2-kube-api-access-j28sv\") pod \"payload-processing-7bb9489676-r5zp7\" (UID: \"40e8212c-49a4-4a50-add6-0ee8a55429b2\") " pod="openshift-ingress/payload-processing-7bb9489676-r5zp7" Nov 03 20:57:01.645886 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:01.645854 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-r5zp7" Nov 03 20:57:01.861218 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:01.861190 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-jjmps"] Nov 03 20:57:01.863276 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:57:01.863247 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f60591a_fc5a_47d9_9899_93963284b5d6.slice/crio-4aea7f5951c5b0b095e698a901b31bb19b80ce35f61613e09b5a174dd543471b WatchSource:0}: Error finding container 4aea7f5951c5b0b095e698a901b31bb19b80ce35f61613e09b5a174dd543471b: Status 404 returned error can't find the container with id 4aea7f5951c5b0b095e698a901b31bb19b80ce35f61613e09b5a174dd543471b Nov 03 20:57:02.010667 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:02.010631 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-r5zp7"] Nov 03 20:57:02.014013 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:57:02.013983 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e8212c_49a4_4a50_add6_0ee8a55429b2.slice/crio-6291d611a4569ac07d7d258b0e7c134c693004e3c48fde7fd314d2c695289ce1 WatchSource:0}: Error finding container 6291d611a4569ac07d7d258b0e7c134c693004e3c48fde7fd314d2c695289ce1: Status 404 returned error can't find the container with id 6291d611a4569ac07d7d258b0e7c134c693004e3c48fde7fd314d2c695289ce1 Nov 03 20:57:02.773307 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:02.773264 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jjmps" event={"ID":"2f60591a-fc5a-47d9-9899-93963284b5d6","Type":"ContainerStarted","Data":"4aea7f5951c5b0b095e698a901b31bb19b80ce35f61613e09b5a174dd543471b"} Nov 03 20:57:02.774579 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:02.774553 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-r5zp7" event={"ID":"40e8212c-49a4-4a50-add6-0ee8a55429b2","Type":"ContainerStarted","Data":"6291d611a4569ac07d7d258b0e7c134c693004e3c48fde7fd314d2c695289ce1"} Nov 03 20:57:05.791617 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:05.791575 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jjmps" event={"ID":"2f60591a-fc5a-47d9-9899-93963284b5d6","Type":"ContainerStarted","Data":"624269bf12c8e3fc860c198e1010d5e2052e217f400ee3639b17f25083ff5fa4"} Nov 03 20:57:05.792090 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:05.791690 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jjmps" Nov 03 20:57:05.793168 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:05.793145 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-r5zp7" event={"ID":"40e8212c-49a4-4a50-add6-0ee8a55429b2","Type":"ContainerStarted","Data":"6b3c1e65feb0bc440eaf188b149fafedc9c6c53b344ee8ac4590d2ee481ecdad"} Nov 03 20:57:05.793275 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:05.793266 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7bb9489676-r5zp7" Nov 03 20:57:05.902427 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:05.902376 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jjmps" podStartSLOduration=1.837819248 podStartE2EDuration="4.902360044s" totalImagesPullingTime="3.064540796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:05.902172543 +0000 UTC m=+520.369951267" watchObservedRunningTime="2026-11-03 20:57:05.902360044 +0000 UTC m=+520.370138768" Nov 03 20:57:06.012138 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:06.012059 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7bb9489676-r5zp7" podStartSLOduration=2.06568852 podStartE2EDuration="5.012047888s" totalImagesPullingTime="2.946359368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:06.011520719 +0000 UTC m=+520.479299437" watchObservedRunningTime="2026-11-03 20:57:06.012047888 +0000 UTC m=+520.479826610" Nov 03 20:57:09.761490 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:09.761460 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-chtmx" Nov 03 20:57:10.119895 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:10.119816 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-b59f75cb9-vv6wb"] Nov 03 20:57:10.123203 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:10.123185 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-b59f75cb9-vv6wb" Nov 03 20:57:10.177487 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:10.177452 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-b59f75cb9-vv6wb"] Nov 03 20:57:10.235850 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:10.235800 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20ef8f90-2fb3-4956-9aa6-cd9578f7c17b-metrics-certs\") pod \"ai-gateway-controller-b59f75cb9-vv6wb\" (UID: \"20ef8f90-2fb3-4956-9aa6-cd9578f7c17b\") " pod="opendatahub/ai-gateway-controller-b59f75cb9-vv6wb" Nov 03 20:57:10.236039 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:10.235959 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2p9x\" (UniqueName: \"kubernetes.io/projected/20ef8f90-2fb3-4956-9aa6-cd9578f7c17b-kube-api-access-j2p9x\") pod \"ai-gateway-controller-b59f75cb9-vv6wb\" (UID: \"20ef8f90-2fb3-4956-9aa6-cd9578f7c17b\") " pod="opendatahub/ai-gateway-controller-b59f75cb9-vv6wb" Nov 03 20:57:10.336931 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:10.336890 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20ef8f90-2fb3-4956-9aa6-cd9578f7c17b-metrics-certs\") pod \"ai-gateway-controller-b59f75cb9-vv6wb\" (UID: \"20ef8f90-2fb3-4956-9aa6-cd9578f7c17b\") " pod="opendatahub/ai-gateway-controller-b59f75cb9-vv6wb" Nov 03 20:57:10.337127 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:10.336949 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2p9x\" (UniqueName: \"kubernetes.io/projected/20ef8f90-2fb3-4956-9aa6-cd9578f7c17b-kube-api-access-j2p9x\") pod \"ai-gateway-controller-b59f75cb9-vv6wb\" (UID: \"20ef8f90-2fb3-4956-9aa6-cd9578f7c17b\") " pod="opendatahub/ai-gateway-controller-b59f75cb9-vv6wb" Nov 03 20:57:10.339258 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:10.339227 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20ef8f90-2fb3-4956-9aa6-cd9578f7c17b-metrics-certs\") pod \"ai-gateway-controller-b59f75cb9-vv6wb\" (UID: \"20ef8f90-2fb3-4956-9aa6-cd9578f7c17b\") " pod="opendatahub/ai-gateway-controller-b59f75cb9-vv6wb" Nov 03 20:57:10.395404 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:10.395340 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2p9x\" (UniqueName: \"kubernetes.io/projected/20ef8f90-2fb3-4956-9aa6-cd9578f7c17b-kube-api-access-j2p9x\") pod \"ai-gateway-controller-b59f75cb9-vv6wb\" (UID: \"20ef8f90-2fb3-4956-9aa6-cd9578f7c17b\") " pod="opendatahub/ai-gateway-controller-b59f75cb9-vv6wb" Nov 03 20:57:10.436379 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:10.436347 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-b59f75cb9-vv6wb" Nov 03 20:57:10.617975 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:57:10.617939 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 20:57:10.798519 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:10.798240 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-b59f75cb9-vv6wb"] Nov 03 20:57:10.801537 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:57:10.801497 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ef8f90_2fb3_4956_9aa6_cd9578f7c17b.slice/crio-128e1d7a67f080dfafc8033b36818363b51aeca7d9169cbf17fb04de3fb1f0f4 WatchSource:0}: Error finding container 128e1d7a67f080dfafc8033b36818363b51aeca7d9169cbf17fb04de3fb1f0f4: Status 404 returned error can't find the container with id 128e1d7a67f080dfafc8033b36818363b51aeca7d9169cbf17fb04de3fb1f0f4 Nov 03 20:57:10.817353 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:10.817317 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-b59f75cb9-vv6wb" event={"ID":"20ef8f90-2fb3-4956-9aa6-cd9578f7c17b","Type":"ContainerStarted","Data":"128e1d7a67f080dfafc8033b36818363b51aeca7d9169cbf17fb04de3fb1f0f4"} Nov 03 20:57:11.823290 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:11.823255 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-b59f75cb9-vv6wb" event={"ID":"20ef8f90-2fb3-4956-9aa6-cd9578f7c17b","Type":"ContainerStarted","Data":"37ac4c02b5f608fba24cd9aa082efdefdc4f6993feb65c2d7b67dbee3297144e"} Nov 03 20:57:11.823707 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:11.823359 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-b59f75cb9-vv6wb" Nov 03 20:57:11.934468 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:11.934362 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-b59f75cb9-vv6wb" podStartSLOduration=1.057418012 podStartE2EDuration="1.934334552s" totalImagesPullingTime="876.91654ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:11.93364951 +0000 UTC m=+526.401428232" watchObservedRunningTime="2026-11-03 20:57:11.934334552 +0000 UTC m=+526.402113276" Nov 03 20:57:16.800561 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:16.800533 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jjmps" Nov 03 20:57:16.802917 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:16.800615 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7bb9489676-r5zp7" Nov 03 20:57:22.830613 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:22.830577 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-b59f75cb9-vv6wb" Nov 03 20:57:22.989791 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:22.989755 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-chtmx"] Nov 03 20:57:22.989982 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:22.989958 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d8dddb676-chtmx" podUID="3fdde323-ed8a-431b-915c-7a5a705903a0" containerName="manager" containerID="cri-o://f38ddd017e8c4fc9bd162285bba5d25ced8493aed40417d9dd93d9ff6166629e" gracePeriod=30 Nov 03 20:57:23.228655 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:23.228624 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-chtmx" Nov 03 20:57:23.352450 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:23.352420 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fdde323-ed8a-431b-915c-7a5a705903a0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fdde323-ed8a-431b-915c-7a5a705903a0-metrics-certs\") pod \"3fdde323-ed8a-431b-915c-7a5a705903a0\" (UID: \"3fdde323-ed8a-431b-915c-7a5a705903a0\") " Nov 03 20:57:23.352626 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:23.352507 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fdde323-ed8a-431b-915c-7a5a705903a0-kube-api-access-5zqgj\" (UniqueName: \"kubernetes.io/projected/3fdde323-ed8a-431b-915c-7a5a705903a0-kube-api-access-5zqgj\") pod \"3fdde323-ed8a-431b-915c-7a5a705903a0\" (UID: \"3fdde323-ed8a-431b-915c-7a5a705903a0\") " Nov 03 20:57:23.354718 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:23.354684 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fdde323-ed8a-431b-915c-7a5a705903a0-metrics-certs" pod "3fdde323-ed8a-431b-915c-7a5a705903a0" (UID: "3fdde323-ed8a-431b-915c-7a5a705903a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:57:23.354835 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:23.354800 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fdde323-ed8a-431b-915c-7a5a705903a0-kube-api-access-5zqgj" pod "3fdde323-ed8a-431b-915c-7a5a705903a0" (UID: "3fdde323-ed8a-431b-915c-7a5a705903a0"). InnerVolumeSpecName "kube-api-access-5zqgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:23.453434 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:23.453392 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fdde323-ed8a-431b-915c-7a5a705903a0-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:23.453434 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:23.453432 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zqgj\" (UniqueName: \"kubernetes.io/projected/3fdde323-ed8a-431b-915c-7a5a705903a0-kube-api-access-5zqgj\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:23.876397 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:23.876363 5615 generic.go:356] "Generic (PLEG): container finished" podID="3fdde323-ed8a-431b-915c-7a5a705903a0" containerID="f38ddd017e8c4fc9bd162285bba5d25ced8493aed40417d9dd93d9ff6166629e" exitCode=0 Nov 03 20:57:23.876802 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:23.876424 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-chtmx" Nov 03 20:57:23.876802 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:23.876454 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-chtmx" event={"ID":"3fdde323-ed8a-431b-915c-7a5a705903a0","Type":"ContainerDied","Data":"f38ddd017e8c4fc9bd162285bba5d25ced8493aed40417d9dd93d9ff6166629e"} Nov 03 20:57:23.876802 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:23.876515 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-chtmx" event={"ID":"3fdde323-ed8a-431b-915c-7a5a705903a0","Type":"ContainerDied","Data":"a99705a9ffc643af6870e5f782086bd4ec13e8f211c76bba66681431608cc9b7"} Nov 03 20:57:23.876802 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:23.876537 5615 scope.go:122] "RemoveContainer" containerID="f38ddd017e8c4fc9bd162285bba5d25ced8493aed40417d9dd93d9ff6166629e" Nov 03 20:57:23.884849 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:23.884830 5615 scope.go:122] "RemoveContainer" containerID="f38ddd017e8c4fc9bd162285bba5d25ced8493aed40417d9dd93d9ff6166629e" Nov 03 20:57:23.885170 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:57:23.885146 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f38ddd017e8c4fc9bd162285bba5d25ced8493aed40417d9dd93d9ff6166629e\": container with ID starting with f38ddd017e8c4fc9bd162285bba5d25ced8493aed40417d9dd93d9ff6166629e not found: ID does not exist" containerID="f38ddd017e8c4fc9bd162285bba5d25ced8493aed40417d9dd93d9ff6166629e" Nov 03 20:57:23.885272 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:23.885178 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f38ddd017e8c4fc9bd162285bba5d25ced8493aed40417d9dd93d9ff6166629e"} err="failed to get container status \"f38ddd017e8c4fc9bd162285bba5d25ced8493aed40417d9dd93d9ff6166629e\": rpc error: code = NotFound desc = could not find container \"f38ddd017e8c4fc9bd162285bba5d25ced8493aed40417d9dd93d9ff6166629e\": container with ID starting with f38ddd017e8c4fc9bd162285bba5d25ced8493aed40417d9dd93d9ff6166629e not found: ID does not exist" Nov 03 20:57:24.032840 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:24.032806 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-chtmx"] Nov 03 20:57:24.079729 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:24.079697 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-chtmx"] Nov 03 20:57:25.649842 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:25.649809 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fdde323-ed8a-431b-915c-7a5a705903a0" path="/var/lib/kubelet/pods/3fdde323-ed8a-431b-915c-7a5a705903a0/volumes" Nov 03 20:57:33.129466 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.129428 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7"] Nov 03 20:57:33.129968 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.129852 5615 cpu_manager.go:414] "containerMap: removing container" podUID="3fdde323-ed8a-431b-915c-7a5a705903a0" containerName="manager" Nov 03 20:57:33.129968 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.129872 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fdde323-ed8a-431b-915c-7a5a705903a0" containerName="manager" Nov 03 20:57:33.129968 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.129956 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fdde323-ed8a-431b-915c-7a5a705903a0" containerName="manager" Nov 03 20:57:33.133268 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.133246 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7" Nov 03 20:57:33.182966 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.182937 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 20:57:33.230227 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.230191 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 20:57:33.230425 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.230273 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-gh4x2\"" Nov 03 20:57:33.230725 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.230704 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 03 20:57:33.231662 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.231638 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7"] Nov 03 20:57:33.241768 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.241748 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42d88418-b5da-43a8-9ff4-edf3bb7a3c56-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7\" (UID: \"42d88418-b5da-43a8-9ff4-edf3bb7a3c56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7" Nov 03 20:57:33.241852 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.241778 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42d88418-b5da-43a8-9ff4-edf3bb7a3c56-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7\" (UID: \"42d88418-b5da-43a8-9ff4-edf3bb7a3c56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7" Nov 03 20:57:33.241852 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.241799 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42d88418-b5da-43a8-9ff4-edf3bb7a3c56-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7\" (UID: \"42d88418-b5da-43a8-9ff4-edf3bb7a3c56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7" Nov 03 20:57:33.241941 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.241923 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckhz2\" (UniqueName: \"kubernetes.io/projected/42d88418-b5da-43a8-9ff4-edf3bb7a3c56-kube-api-access-ckhz2\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7\" (UID: \"42d88418-b5da-43a8-9ff4-edf3bb7a3c56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7" Nov 03 20:57:33.242005 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.241993 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42d88418-b5da-43a8-9ff4-edf3bb7a3c56-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7\" (UID: \"42d88418-b5da-43a8-9ff4-edf3bb7a3c56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7" Nov 03 20:57:33.242042 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.242019 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42d88418-b5da-43a8-9ff4-edf3bb7a3c56-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7\" (UID: \"42d88418-b5da-43a8-9ff4-edf3bb7a3c56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7" Nov 03 20:57:33.343481 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.343446 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42d88418-b5da-43a8-9ff4-edf3bb7a3c56-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7\" (UID: \"42d88418-b5da-43a8-9ff4-edf3bb7a3c56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7" Nov 03 20:57:33.343481 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.343487 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42d88418-b5da-43a8-9ff4-edf3bb7a3c56-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7\" (UID: \"42d88418-b5da-43a8-9ff4-edf3bb7a3c56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7" Nov 03 20:57:33.343735 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.343502 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42d88418-b5da-43a8-9ff4-edf3bb7a3c56-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7\" (UID: \"42d88418-b5da-43a8-9ff4-edf3bb7a3c56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7" Nov 03 20:57:33.343735 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.343568 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckhz2\" (UniqueName: \"kubernetes.io/projected/42d88418-b5da-43a8-9ff4-edf3bb7a3c56-kube-api-access-ckhz2\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7\" (UID: \"42d88418-b5da-43a8-9ff4-edf3bb7a3c56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7" Nov 03 20:57:33.343735 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.343641 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42d88418-b5da-43a8-9ff4-edf3bb7a3c56-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7\" (UID: \"42d88418-b5da-43a8-9ff4-edf3bb7a3c56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7" Nov 03 20:57:33.343735 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.343677 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42d88418-b5da-43a8-9ff4-edf3bb7a3c56-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7\" (UID: \"42d88418-b5da-43a8-9ff4-edf3bb7a3c56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7" Nov 03 20:57:33.343944 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.343925 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42d88418-b5da-43a8-9ff4-edf3bb7a3c56-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7\" (UID: \"42d88418-b5da-43a8-9ff4-edf3bb7a3c56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7" Nov 03 20:57:33.344004 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.343979 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42d88418-b5da-43a8-9ff4-edf3bb7a3c56-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7\" (UID: \"42d88418-b5da-43a8-9ff4-edf3bb7a3c56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7" Nov 03 20:57:33.344133 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.344064 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42d88418-b5da-43a8-9ff4-edf3bb7a3c56-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7\" (UID: \"42d88418-b5da-43a8-9ff4-edf3bb7a3c56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7" Nov 03 20:57:33.345960 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.345936 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42d88418-b5da-43a8-9ff4-edf3bb7a3c56-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7\" (UID: \"42d88418-b5da-43a8-9ff4-edf3bb7a3c56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7" Nov 03 20:57:33.346152 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.346134 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42d88418-b5da-43a8-9ff4-edf3bb7a3c56-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7\" (UID: \"42d88418-b5da-43a8-9ff4-edf3bb7a3c56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7" Nov 03 20:57:33.402680 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.402605 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckhz2\" (UniqueName: \"kubernetes.io/projected/42d88418-b5da-43a8-9ff4-edf3bb7a3c56-kube-api-access-ckhz2\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7\" (UID: \"42d88418-b5da-43a8-9ff4-edf3bb7a3c56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7" Nov 03 20:57:33.444786 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.444740 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7" Nov 03 20:57:33.809063 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.809035 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7"] Nov 03 20:57:33.811602 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:57:33.811565 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d88418_b5da_43a8_9ff4_edf3bb7a3c56.slice/crio-308d2302d4d5e57a58848a094e2ca3d85777cfffb7cffaf0e81e6f16dd90a8a9 WatchSource:0}: Error finding container 308d2302d4d5e57a58848a094e2ca3d85777cfffb7cffaf0e81e6f16dd90a8a9: Status 404 returned error can't find the container with id 308d2302d4d5e57a58848a094e2ca3d85777cfffb7cffaf0e81e6f16dd90a8a9 Nov 03 20:57:33.923557 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:33.923519 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7" event={"ID":"42d88418-b5da-43a8-9ff4-edf3bb7a3c56","Type":"ContainerStarted","Data":"308d2302d4d5e57a58848a094e2ca3d85777cfffb7cffaf0e81e6f16dd90a8a9"} Nov 03 20:57:36.938228 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:36.938184 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7" event={"ID":"42d88418-b5da-43a8-9ff4-edf3bb7a3c56","Type":"ContainerStarted","Data":"62caae050d8421c0ee7a233517b0d750bc709e09e11482205724bf5895541de2"} Nov 03 20:57:37.051931 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:37.051868 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7" podStartSLOduration=1.922836261 podStartE2EDuration="4.051854407s" totalImagesPullingTime="2.129018146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:37.050644794 +0000 UTC m=+551.518423517" watchObservedRunningTime="2026-11-03 20:57:37.051854407 +0000 UTC m=+551.519633127" Nov 03 20:57:37.445581 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:37.445539 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7" Nov 03 20:57:40.618197 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:57:40.618158 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 20:57:43.050426 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:43.050385 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-hnr66"] Nov 03 20:57:43.053950 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:43.053931 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hnr66" Nov 03 20:57:43.150516 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:43.150487 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 03 20:57:43.152011 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:43.151983 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-hnr66"] Nov 03 20:57:43.244041 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:43.244003 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dccff396-4195-41f2-ada5-110b2c0ab303-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hnr66\" (UID: \"dccff396-4195-41f2-ada5-110b2c0ab303\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hnr66" Nov 03 20:57:43.244041 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:43.244037 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dccff396-4195-41f2-ada5-110b2c0ab303-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hnr66\" (UID: \"dccff396-4195-41f2-ada5-110b2c0ab303\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hnr66" Nov 03 20:57:43.244290 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:43.244148 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dccff396-4195-41f2-ada5-110b2c0ab303-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hnr66\" (UID: \"dccff396-4195-41f2-ada5-110b2c0ab303\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hnr66" Nov 03 20:57:43.244290 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:43.244181 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kntm\" (UniqueName: \"kubernetes.io/projected/dccff396-4195-41f2-ada5-110b2c0ab303-kube-api-access-8kntm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hnr66\" (UID: \"dccff396-4195-41f2-ada5-110b2c0ab303\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hnr66" Nov 03 20:57:43.244290 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:43.244200 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dccff396-4195-41f2-ada5-110b2c0ab303-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hnr66\" (UID: \"dccff396-4195-41f2-ada5-110b2c0ab303\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hnr66" Nov 03 20:57:43.244290 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:43.244223 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dccff396-4195-41f2-ada5-110b2c0ab303-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hnr66\" (UID: \"dccff396-4195-41f2-ada5-110b2c0ab303\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hnr66" Nov 03 20:57:43.345147 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:43.345037 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dccff396-4195-41f2-ada5-110b2c0ab303-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hnr66\" (UID: \"dccff396-4195-41f2-ada5-110b2c0ab303\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hnr66" Nov 03 20:57:43.345147 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:43.345091 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dccff396-4195-41f2-ada5-110b2c0ab303-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hnr66\" (UID: \"dccff396-4195-41f2-ada5-110b2c0ab303\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hnr66" Nov 03 20:57:43.345147 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:43.345148 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dccff396-4195-41f2-ada5-110b2c0ab303-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hnr66\" (UID: \"dccff396-4195-41f2-ada5-110b2c0ab303\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hnr66" Nov 03 20:57:43.345426 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:43.345185 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kntm\" (UniqueName: \"kubernetes.io/projected/dccff396-4195-41f2-ada5-110b2c0ab303-kube-api-access-8kntm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hnr66\" (UID: \"dccff396-4195-41f2-ada5-110b2c0ab303\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hnr66" Nov 03 20:57:43.345426 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:43.345216 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dccff396-4195-41f2-ada5-110b2c0ab303-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hnr66\" (UID: \"dccff396-4195-41f2-ada5-110b2c0ab303\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hnr66" Nov 03 20:57:43.345426 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:43.345251 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dccff396-4195-41f2-ada5-110b2c0ab303-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hnr66\" (UID: \"dccff396-4195-41f2-ada5-110b2c0ab303\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hnr66" Nov 03 20:57:43.345590 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:43.345537 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dccff396-4195-41f2-ada5-110b2c0ab303-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hnr66\" (UID: \"dccff396-4195-41f2-ada5-110b2c0ab303\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hnr66" Nov 03 20:57:43.345590 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:43.345570 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dccff396-4195-41f2-ada5-110b2c0ab303-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hnr66\" (UID: \"dccff396-4195-41f2-ada5-110b2c0ab303\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hnr66" Nov 03 20:57:43.345699 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:43.345677 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dccff396-4195-41f2-ada5-110b2c0ab303-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hnr66\" (UID: \"dccff396-4195-41f2-ada5-110b2c0ab303\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hnr66" Nov 03 20:57:43.347570 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:43.347545 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dccff396-4195-41f2-ada5-110b2c0ab303-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hnr66\" (UID: \"dccff396-4195-41f2-ada5-110b2c0ab303\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hnr66" Nov 03 20:57:43.347685 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:43.347664 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dccff396-4195-41f2-ada5-110b2c0ab303-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hnr66\" (UID: \"dccff396-4195-41f2-ada5-110b2c0ab303\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hnr66" Nov 03 20:57:43.406245 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:43.406215 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kntm\" (UniqueName: \"kubernetes.io/projected/dccff396-4195-41f2-ada5-110b2c0ab303-kube-api-access-8kntm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hnr66\" (UID: \"dccff396-4195-41f2-ada5-110b2c0ab303\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hnr66" Nov 03 20:57:43.445970 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:43.445926 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7" Nov 03 20:57:43.458591 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:43.458559 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7" Nov 03 20:57:43.666708 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:43.666633 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hnr66" Nov 03 20:57:43.981332 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:43.981246 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7" Nov 03 20:57:44.031664 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:44.031642 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-hnr66"] Nov 03 20:57:44.034500 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:57:44.034476 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddccff396_4195_41f2_ada5_110b2c0ab303.slice/crio-7103a1b4060ad8ca6345dbebd626615c4ce74824c6ea6c4e29dae045e19e542a WatchSource:0}: Error finding container 7103a1b4060ad8ca6345dbebd626615c4ce74824c6ea6c4e29dae045e19e542a: Status 404 returned error can't find the container with id 7103a1b4060ad8ca6345dbebd626615c4ce74824c6ea6c4e29dae045e19e542a Nov 03 20:57:44.976345 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:44.976250 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hnr66" event={"ID":"dccff396-4195-41f2-ada5-110b2c0ab303","Type":"ContainerStarted","Data":"fc4c0635910ff83982effe1fdd9d499eddf37f623b44295481c5b282eed2bc43"} Nov 03 20:57:44.976345 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:44.976326 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hnr66" event={"ID":"dccff396-4195-41f2-ada5-110b2c0ab303","Type":"ContainerStarted","Data":"7103a1b4060ad8ca6345dbebd626615c4ce74824c6ea6c4e29dae045e19e542a"} Nov 03 20:57:45.088250 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:45.088195 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hnr66" podStartSLOduration=1.455257988 podStartE2EDuration="2.088180825s" totalImagesPullingTime="632.922837ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:45.0879041 +0000 UTC m=+559.555682823" watchObservedRunningTime="2026-11-03 20:57:45.088180825 +0000 UTC m=+559.555959548" Nov 03 20:57:45.666833 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:45.666781 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hnr66" Nov 03 20:57:50.371770 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:50.371732 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f"] Nov 03 20:57:50.375657 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:50.375630 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f" Nov 03 20:57:50.432940 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:50.432888 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46bf6611-811b-4e94-85c2-f6ad5871f3e8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f\" (UID: \"46bf6611-811b-4e94-85c2-f6ad5871f3e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f" Nov 03 20:57:50.433145 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:50.432950 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lkwp\" (UniqueName: \"kubernetes.io/projected/46bf6611-811b-4e94-85c2-f6ad5871f3e8-kube-api-access-8lkwp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f\" (UID: \"46bf6611-811b-4e94-85c2-f6ad5871f3e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f" Nov 03 20:57:50.433145 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:50.432987 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46bf6611-811b-4e94-85c2-f6ad5871f3e8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f\" (UID: \"46bf6611-811b-4e94-85c2-f6ad5871f3e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f" Nov 03 20:57:50.433145 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:50.433018 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46bf6611-811b-4e94-85c2-f6ad5871f3e8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f\" (UID: \"46bf6611-811b-4e94-85c2-f6ad5871f3e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f" Nov 03 20:57:50.433351 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:50.433172 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46bf6611-811b-4e94-85c2-f6ad5871f3e8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f\" (UID: \"46bf6611-811b-4e94-85c2-f6ad5871f3e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f" Nov 03 20:57:50.433351 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:50.433202 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46bf6611-811b-4e94-85c2-f6ad5871f3e8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f\" (UID: \"46bf6611-811b-4e94-85c2-f6ad5871f3e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f" Nov 03 20:57:50.472804 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:50.472763 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 03 20:57:50.475709 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:50.475672 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f"] Nov 03 20:57:50.534717 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:50.534675 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46bf6611-811b-4e94-85c2-f6ad5871f3e8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f\" (UID: \"46bf6611-811b-4e94-85c2-f6ad5871f3e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f" Nov 03 20:57:50.534882 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:50.534799 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46bf6611-811b-4e94-85c2-f6ad5871f3e8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f\" (UID: \"46bf6611-811b-4e94-85c2-f6ad5871f3e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f" Nov 03 20:57:50.534882 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:50.534827 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46bf6611-811b-4e94-85c2-f6ad5871f3e8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f\" (UID: \"46bf6611-811b-4e94-85c2-f6ad5871f3e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f" Nov 03 20:57:50.534882 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:50.534846 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lkwp\" (UniqueName: \"kubernetes.io/projected/46bf6611-811b-4e94-85c2-f6ad5871f3e8-kube-api-access-8lkwp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f\" (UID: \"46bf6611-811b-4e94-85c2-f6ad5871f3e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f" Nov 03 20:57:50.534882 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:50.534865 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46bf6611-811b-4e94-85c2-f6ad5871f3e8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f\" (UID: \"46bf6611-811b-4e94-85c2-f6ad5871f3e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f" Nov 03 20:57:50.534882 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:50.534882 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46bf6611-811b-4e94-85c2-f6ad5871f3e8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f\" (UID: \"46bf6611-811b-4e94-85c2-f6ad5871f3e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f" Nov 03 20:57:50.535278 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:50.535246 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46bf6611-811b-4e94-85c2-f6ad5871f3e8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f\" (UID: \"46bf6611-811b-4e94-85c2-f6ad5871f3e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f" Nov 03 20:57:50.535381 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:50.535283 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46bf6611-811b-4e94-85c2-f6ad5871f3e8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f\" (UID: \"46bf6611-811b-4e94-85c2-f6ad5871f3e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f" Nov 03 20:57:50.535381 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:50.535314 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46bf6611-811b-4e94-85c2-f6ad5871f3e8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f\" (UID: \"46bf6611-811b-4e94-85c2-f6ad5871f3e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f" Nov 03 20:57:50.537258 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:50.537228 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46bf6611-811b-4e94-85c2-f6ad5871f3e8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f\" (UID: \"46bf6611-811b-4e94-85c2-f6ad5871f3e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f" Nov 03 20:57:50.537389 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:50.537374 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46bf6611-811b-4e94-85c2-f6ad5871f3e8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f\" (UID: \"46bf6611-811b-4e94-85c2-f6ad5871f3e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f" Nov 03 20:57:50.595107 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:50.595046 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lkwp\" (UniqueName: \"kubernetes.io/projected/46bf6611-811b-4e94-85c2-f6ad5871f3e8-kube-api-access-8lkwp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f\" (UID: \"46bf6611-811b-4e94-85c2-f6ad5871f3e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f" Nov 03 20:57:50.688794 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:50.688700 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f" Nov 03 20:57:51.058466 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:51.058438 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f"] Nov 03 20:57:51.060678 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:57:51.060650 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46bf6611_811b_4e94_85c2_f6ad5871f3e8.slice/crio-9473de44fd40fe530625ab34ba8f7797478429fa6ef34e27dec3b456c2d0730a WatchSource:0}: Error finding container 9473de44fd40fe530625ab34ba8f7797478429fa6ef34e27dec3b456c2d0730a: Status 404 returned error can't find the container with id 9473de44fd40fe530625ab34ba8f7797478429fa6ef34e27dec3b456c2d0730a Nov 03 20:57:52.015683 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:52.015651 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f" event={"ID":"46bf6611-811b-4e94-85c2-f6ad5871f3e8","Type":"ContainerStarted","Data":"90a472963b7cd7b62d90918501c29c3c488a0a35abc74bb83ed94c522dfa2c2e"} Nov 03 20:57:52.016093 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:52.015702 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f" event={"ID":"46bf6611-811b-4e94-85c2-f6ad5871f3e8","Type":"ContainerStarted","Data":"9473de44fd40fe530625ab34ba8f7797478429fa6ef34e27dec3b456c2d0730a"} Nov 03 20:57:52.127969 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:52.127913 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f" podStartSLOduration=1.488370528 podStartE2EDuration="2.127898794s" totalImagesPullingTime="639.528266ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:52.127462783 +0000 UTC m=+566.595241506" watchObservedRunningTime="2026-11-03 20:57:52.127898794 +0000 UTC m=+566.595677517" Nov 03 20:57:52.688950 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:52.688912 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f" Nov 03 20:57:53.666838 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:53.666808 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hnr66" Nov 03 20:57:53.679138 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:53.679114 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hnr66" Nov 03 20:57:54.037338 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:54.037307 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hnr66" Nov 03 20:57:59.643223 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:59.643188 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v"] Nov 03 20:57:59.647552 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:59.647525 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v" Nov 03 20:57:59.733681 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:59.733649 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66f8b152-0f96-4ab7-b6de-a7beb77d81c6-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v\" (UID: \"66f8b152-0f96-4ab7-b6de-a7beb77d81c6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v" Nov 03 20:57:59.733842 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:59.733685 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66f8b152-0f96-4ab7-b6de-a7beb77d81c6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v\" (UID: \"66f8b152-0f96-4ab7-b6de-a7beb77d81c6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v" Nov 03 20:57:59.733842 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:59.733717 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66f8b152-0f96-4ab7-b6de-a7beb77d81c6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v\" (UID: \"66f8b152-0f96-4ab7-b6de-a7beb77d81c6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v" Nov 03 20:57:59.733970 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:59.733924 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66f8b152-0f96-4ab7-b6de-a7beb77d81c6-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v\" (UID: \"66f8b152-0f96-4ab7-b6de-a7beb77d81c6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v" Nov 03 20:57:59.734043 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:59.734025 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66f8b152-0f96-4ab7-b6de-a7beb77d81c6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v\" (UID: \"66f8b152-0f96-4ab7-b6de-a7beb77d81c6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v" Nov 03 20:57:59.734116 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:59.734057 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn7js\" (UniqueName: \"kubernetes.io/projected/66f8b152-0f96-4ab7-b6de-a7beb77d81c6-kube-api-access-nn7js\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v\" (UID: \"66f8b152-0f96-4ab7-b6de-a7beb77d81c6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v" Nov 03 20:57:59.744042 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:59.744011 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 03 20:57:59.745451 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:59.745426 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v"] Nov 03 20:57:59.834909 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:59.834875 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66f8b152-0f96-4ab7-b6de-a7beb77d81c6-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v\" (UID: \"66f8b152-0f96-4ab7-b6de-a7beb77d81c6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v" Nov 03 20:57:59.835181 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:59.834939 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66f8b152-0f96-4ab7-b6de-a7beb77d81c6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v\" (UID: \"66f8b152-0f96-4ab7-b6de-a7beb77d81c6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v" Nov 03 20:57:59.835181 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:59.834964 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn7js\" (UniqueName: \"kubernetes.io/projected/66f8b152-0f96-4ab7-b6de-a7beb77d81c6-kube-api-access-nn7js\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v\" (UID: \"66f8b152-0f96-4ab7-b6de-a7beb77d81c6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v" Nov 03 20:57:59.835181 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:59.835133 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66f8b152-0f96-4ab7-b6de-a7beb77d81c6-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v\" (UID: \"66f8b152-0f96-4ab7-b6de-a7beb77d81c6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v" Nov 03 20:57:59.835181 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:59.835182 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66f8b152-0f96-4ab7-b6de-a7beb77d81c6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v\" (UID: \"66f8b152-0f96-4ab7-b6de-a7beb77d81c6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v" Nov 03 20:57:59.835398 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:59.835221 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66f8b152-0f96-4ab7-b6de-a7beb77d81c6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v\" (UID: \"66f8b152-0f96-4ab7-b6de-a7beb77d81c6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v" Nov 03 20:57:59.835439 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:59.835425 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66f8b152-0f96-4ab7-b6de-a7beb77d81c6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v\" (UID: \"66f8b152-0f96-4ab7-b6de-a7beb77d81c6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v" Nov 03 20:57:59.835545 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:59.835513 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66f8b152-0f96-4ab7-b6de-a7beb77d81c6-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v\" (UID: \"66f8b152-0f96-4ab7-b6de-a7beb77d81c6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v" Nov 03 20:57:59.835623 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:59.835599 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66f8b152-0f96-4ab7-b6de-a7beb77d81c6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v\" (UID: \"66f8b152-0f96-4ab7-b6de-a7beb77d81c6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v" Nov 03 20:57:59.837282 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:59.837259 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66f8b152-0f96-4ab7-b6de-a7beb77d81c6-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v\" (UID: \"66f8b152-0f96-4ab7-b6de-a7beb77d81c6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v" Nov 03 20:57:59.837549 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:59.837520 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66f8b152-0f96-4ab7-b6de-a7beb77d81c6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v\" (UID: \"66f8b152-0f96-4ab7-b6de-a7beb77d81c6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v" Nov 03 20:57:59.893991 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:59.893912 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn7js\" (UniqueName: \"kubernetes.io/projected/66f8b152-0f96-4ab7-b6de-a7beb77d81c6-kube-api-access-nn7js\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v\" (UID: \"66f8b152-0f96-4ab7-b6de-a7beb77d81c6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v" Nov 03 20:57:59.960338 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:57:59.960297 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v" Nov 03 20:58:00.326246 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:00.326220 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v"] Nov 03 20:58:00.328563 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:58:00.328526 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f8b152_0f96_4ab7_b6de_a7beb77d81c6.slice/crio-cfb88bfc8e857a0f8fc3c31a233dd32ea68c4253709296b935755a00c4ca707f WatchSource:0}: Error finding container cfb88bfc8e857a0f8fc3c31a233dd32ea68c4253709296b935755a00c4ca707f: Status 404 returned error can't find the container with id cfb88bfc8e857a0f8fc3c31a233dd32ea68c4253709296b935755a00c4ca707f Nov 03 20:58:00.689888 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:00.689805 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f" Nov 03 20:58:00.702546 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:00.702521 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f" Nov 03 20:58:01.063390 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:01.063347 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v" event={"ID":"66f8b152-0f96-4ab7-b6de-a7beb77d81c6","Type":"ContainerStarted","Data":"7f2fc0c9a6270c1b299fbf9a1d3e098c3b6fb28934579bc087bc82ffbc82c552"} Nov 03 20:58:01.063566 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:01.063407 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v" event={"ID":"66f8b152-0f96-4ab7-b6de-a7beb77d81c6","Type":"ContainerStarted","Data":"cfb88bfc8e857a0f8fc3c31a233dd32ea68c4253709296b935755a00c4ca707f"} Nov 03 20:58:01.075170 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:01.075140 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f" Nov 03 20:58:01.175437 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:01.175387 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v" podStartSLOduration=1.569128567 podStartE2EDuration="2.175372797s" totalImagesPullingTime="606.24423ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:01.174676938 +0000 UTC m=+575.642455662" watchObservedRunningTime="2026-11-03 20:58:01.175372797 +0000 UTC m=+575.643151519" Nov 03 20:58:01.960670 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:01.960627 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v" Nov 03 20:58:09.961299 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:09.961257 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v" Nov 03 20:58:09.974127 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:09.974095 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v" Nov 03 20:58:10.117135 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:10.117105 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v" Nov 03 20:58:10.618798 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:58:10.618755 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 20:58:11.018850 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.018813 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2"] Nov 03 20:58:11.022313 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.022288 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:11.072267 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.072235 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4cjrg\"" Nov 03 20:58:11.121275 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.121239 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2"] Nov 03 20:58:11.155585 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.155543 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/889d1dcf-c570-4151-9884-ddc9962a4c0e-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4zxf2\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:11.155762 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.155598 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4zxf2\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:11.155762 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.155689 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4zxf2\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:11.155762 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.155736 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/889d1dcf-c570-4151-9884-ddc9962a4c0e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4zxf2\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:11.155865 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.155781 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4zxf2\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:11.155865 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.155799 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/889d1dcf-c570-4151-9884-ddc9962a4c0e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4zxf2\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:11.155942 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.155923 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4qzg\" (UniqueName: \"kubernetes.io/projected/889d1dcf-c570-4151-9884-ddc9962a4c0e-kube-api-access-f4qzg\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4zxf2\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:11.155980 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.155952 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4zxf2\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:11.155980 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.155971 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4zxf2\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:11.256746 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.256705 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/889d1dcf-c570-4151-9884-ddc9962a4c0e-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4zxf2\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:11.256937 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.256758 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4zxf2\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:11.256937 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.256785 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4zxf2\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:11.256937 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.256802 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/889d1dcf-c570-4151-9884-ddc9962a4c0e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4zxf2\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:11.256937 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.256848 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4zxf2\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:11.256937 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.256863 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/889d1dcf-c570-4151-9884-ddc9962a4c0e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4zxf2\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:11.257237 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.257070 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4qzg\" (UniqueName: \"kubernetes.io/projected/889d1dcf-c570-4151-9884-ddc9962a4c0e-kube-api-access-f4qzg\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4zxf2\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:11.257237 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.257123 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4zxf2\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:11.257237 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.257148 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4zxf2\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:11.257374 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.257240 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4zxf2\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:11.257374 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.257325 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4zxf2\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:11.257588 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.257563 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4zxf2\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:11.257709 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.257687 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4zxf2\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:11.257923 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.257904 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/889d1dcf-c570-4151-9884-ddc9962a4c0e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4zxf2\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:11.259333 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.259310 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4zxf2\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:11.259701 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.259675 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/889d1dcf-c570-4151-9884-ddc9962a4c0e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4zxf2\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:11.314950 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.314884 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/889d1dcf-c570-4151-9884-ddc9962a4c0e-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4zxf2\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:11.315103 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.314959 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4qzg\" (UniqueName: \"kubernetes.io/projected/889d1dcf-c570-4151-9884-ddc9962a4c0e-kube-api-access-f4qzg\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4zxf2\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:11.334480 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.334454 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:11.747356 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:11.747333 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2"] Nov 03 20:58:11.749264 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:58:11.749239 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod889d1dcf_c570_4151_9884_ddc9962a4c0e.slice/crio-4e1606c9668002adb0fc480cc42e50e38c7c3d56b6dc1f74ed07d050cba1b9ef WatchSource:0}: Error finding container 4e1606c9668002adb0fc480cc42e50e38c7c3d56b6dc1f74ed07d050cba1b9ef: Status 404 returned error can't find the container with id 4e1606c9668002adb0fc480cc42e50e38c7c3d56b6dc1f74ed07d050cba1b9ef Nov 03 20:58:12.118382 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:12.118300 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" event={"ID":"889d1dcf-c570-4151-9884-ddc9962a4c0e","Type":"ContainerStarted","Data":"4e1606c9668002adb0fc480cc42e50e38c7c3d56b6dc1f74ed07d050cba1b9ef"} Nov 03 20:58:15.807730 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:15.807693 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:58:15.808032 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:15.807765 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:58:15.808032 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:15.807792 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:58:16.139424 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:16.139339 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" event={"ID":"889d1dcf-c570-4151-9884-ddc9962a4c0e","Type":"ContainerStarted","Data":"c91eec024676eb3ba61073787252ef5cb26556f7d082bc7f891d93a4da9d2c81"} Nov 03 20:58:16.307098 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:16.305498 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" podStartSLOduration=2.249956908 podStartE2EDuration="6.305473807s" totalImagesPullingTime="4.055516899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:16.303609329 +0000 UTC m=+590.771388054" watchObservedRunningTime="2026-11-03 20:58:16.305473807 +0000 UTC m=+590.773252532" Nov 03 20:58:16.335276 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:16.335233 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:16.336939 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:16.336911 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 20:58:16.337106 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:16.336970 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" podUID="889d1dcf-c570-4151-9884-ddc9962a4c0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 20:58:17.335297 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:17.335261 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 20:58:17.335669 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:17.335317 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" podUID="889d1dcf-c570-4151-9884-ddc9962a4c0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 20:58:18.335542 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:18.335502 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 20:58:18.335948 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:18.335578 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" podUID="889d1dcf-c570-4151-9884-ddc9962a4c0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 20:58:19.335406 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:19.335369 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 20:58:19.335589 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:19.335422 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" podUID="889d1dcf-c570-4151-9884-ddc9962a4c0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 20:58:20.335295 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:20.335264 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 20:58:20.335462 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:20.335314 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" podUID="889d1dcf-c570-4151-9884-ddc9962a4c0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 20:58:21.335095 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:21.335033 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 20:58:21.335573 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:21.335127 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" podUID="889d1dcf-c570-4151-9884-ddc9962a4c0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 20:58:22.335527 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:22.335493 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 20:58:22.336051 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:22.335547 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" podUID="889d1dcf-c570-4151-9884-ddc9962a4c0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 20:58:23.335767 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:23.335734 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 20:58:23.336329 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:23.335793 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" podUID="889d1dcf-c570-4151-9884-ddc9962a4c0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 20:58:24.339480 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:24.339442 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:24.339887 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:24.339740 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:24.340368 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:24.340348 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 20:58:33.319502 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:33.319464 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5696f576-mwgpm"] Nov 03 20:58:33.320015 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:33.319686 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5696f576-mwgpm" podUID="8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3" containerName="authorino" containerID="cri-o://ee93e11ed029b37f8297e7972989f1014ece5c7af01dec7768ba8f58761e14d7" gracePeriod=30 Nov 03 20:58:33.565402 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:33.565377 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5696f576-mwgpm" Nov 03 20:58:33.687058 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:33.686982 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3-kube-api-access-ml72g\" (UniqueName: \"kubernetes.io/projected/8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3-kube-api-access-ml72g\") pod \"8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3\" (UID: \"8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3\") " Nov 03 20:58:33.687244 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:33.687097 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3-tls-cert\" (UniqueName: \"kubernetes.io/secret/8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3-tls-cert\") pod \"8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3\" (UID: \"8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3\") " Nov 03 20:58:33.689126 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:33.689094 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3-kube-api-access-ml72g" pod "8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3" (UID: "8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3"). InnerVolumeSpecName "kube-api-access-ml72g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:33.701059 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:33.701027 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3-tls-cert" pod "8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3" (UID: "8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:58:33.788490 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:33.788449 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml72g\" (UniqueName: \"kubernetes.io/projected/8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3-kube-api-access-ml72g\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:33.788490 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:33.788483 5615 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3-tls-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:34.227593 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:34.227554 5615 generic.go:356] "Generic (PLEG): container finished" podID="8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3" containerID="ee93e11ed029b37f8297e7972989f1014ece5c7af01dec7768ba8f58761e14d7" exitCode=0 Nov 03 20:58:34.227783 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:34.227598 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5696f576-mwgpm" Nov 03 20:58:34.227783 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:34.227639 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5696f576-mwgpm" event={"ID":"8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3","Type":"ContainerDied","Data":"ee93e11ed029b37f8297e7972989f1014ece5c7af01dec7768ba8f58761e14d7"} Nov 03 20:58:34.227783 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:34.227686 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5696f576-mwgpm" event={"ID":"8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3","Type":"ContainerDied","Data":"01edb3254cae24d4d514699de686647e1dfc8251043fa0bb47bf1c9a42dda0e5"} Nov 03 20:58:34.227783 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:34.227708 5615 scope.go:122] "RemoveContainer" containerID="ee93e11ed029b37f8297e7972989f1014ece5c7af01dec7768ba8f58761e14d7" Nov 03 20:58:34.236682 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:34.236663 5615 scope.go:122] "RemoveContainer" containerID="ee93e11ed029b37f8297e7972989f1014ece5c7af01dec7768ba8f58761e14d7" Nov 03 20:58:34.236942 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:58:34.236924 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee93e11ed029b37f8297e7972989f1014ece5c7af01dec7768ba8f58761e14d7\": container with ID starting with ee93e11ed029b37f8297e7972989f1014ece5c7af01dec7768ba8f58761e14d7 not found: ID does not exist" containerID="ee93e11ed029b37f8297e7972989f1014ece5c7af01dec7768ba8f58761e14d7" Nov 03 20:58:34.236990 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:34.236950 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee93e11ed029b37f8297e7972989f1014ece5c7af01dec7768ba8f58761e14d7"} err="failed to get container status \"ee93e11ed029b37f8297e7972989f1014ece5c7af01dec7768ba8f58761e14d7\": rpc error: code = NotFound desc = could not find container \"ee93e11ed029b37f8297e7972989f1014ece5c7af01dec7768ba8f58761e14d7\": container with ID starting with ee93e11ed029b37f8297e7972989f1014ece5c7af01dec7768ba8f58761e14d7 not found: ID does not exist" Nov 03 20:58:34.382467 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:34.382433 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5696f576-mwgpm"] Nov 03 20:58:34.429433 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:34.429398 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5696f576-mwgpm"] Nov 03 20:58:35.651144 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:58:35.651112 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3" path="/var/lib/kubelet/pods/8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3/volumes" Nov 03 20:58:40.619144 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:58:40.619065 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 20:59:04.414151 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.414113 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc"] Nov 03 20:59:04.414619 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.414551 5615 cpu_manager.go:414] "containerMap: removing container" podUID="8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3" containerName="authorino" Nov 03 20:59:04.414619 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.414563 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3" containerName="authorino" Nov 03 20:59:04.414702 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.414632 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d9784e5-3fcf-40d4-9416-cdcf6cf1acb3" containerName="authorino" Nov 03 20:59:04.420552 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.420528 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:04.490264 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.490224 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-workload-socket\") pod \"e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:04.490436 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.490272 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istio-data\") pod \"e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:04.490436 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.490334 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istio-envoy\") pod \"e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:04.490436 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.490371 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istio-podinfo\") pod \"e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:04.490558 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.490467 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istio-token\") pod \"e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:04.490558 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.490504 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgkpc\" (UniqueName: \"kubernetes.io/projected/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-kube-api-access-lgkpc\") pod \"e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:04.490558 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.490529 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-workload-certs\") pod \"e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:04.490662 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.490565 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istiod-ca-cert\") pod \"e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:04.490662 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.490601 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-credential-socket\") pod \"e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:04.537165 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.537132 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc"] Nov 03 20:59:04.553761 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.553732 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-10bfccd7-openshift-default-dockercfg-zckrq\"" Nov 03 20:59:04.591127 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.591058 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istio-podinfo\") pod \"e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:04.591402 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.591181 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istio-token\") pod \"e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:04.591402 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.591268 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgkpc\" (UniqueName: \"kubernetes.io/projected/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-kube-api-access-lgkpc\") pod \"e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:04.591402 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.591304 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-workload-certs\") pod \"e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:04.591402 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.591353 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istiod-ca-cert\") pod \"e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:04.591622 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.591411 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-credential-socket\") pod \"e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:04.591622 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.591445 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-workload-socket\") pod \"e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:04.591622 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.591475 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istio-data\") pod \"e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:04.591622 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.591506 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istio-envoy\") pod \"e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:04.591912 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.591886 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-workload-socket\") pod \"e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:04.591980 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.591919 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istio-data\") pod \"e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:04.592204 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.592175 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-credential-socket\") pod \"e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:04.592328 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.592298 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istiod-ca-cert\") pod \"e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:04.592390 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.592367 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-workload-certs\") pod \"e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:04.593835 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.593812 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istio-podinfo\") pod \"e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:04.594144 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.594124 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istio-envoy\") pod \"e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:04.649698 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.649665 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istio-token\") pod \"e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:04.649862 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.649769 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgkpc\" (UniqueName: \"kubernetes.io/projected/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-kube-api-access-lgkpc\") pod \"e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:04.733127 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.733095 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:04.772529 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.772494 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m"] Nov 03 20:59:04.776643 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.776618 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:04.796091 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.796048 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-workload-certs\") pod \"e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:04.796200 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.796130 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-istio-data\") pod \"e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:04.796200 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.796159 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c22d0b-c50f-4f98-a101-846557ad04f6-istiod-ca-cert\") pod \"e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:04.796200 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.796187 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-workload-socket\") pod \"e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:04.796316 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.796250 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45bx4\" (UniqueName: \"kubernetes.io/projected/17c22d0b-c50f-4f98-a101-846557ad04f6-kube-api-access-45bx4\") pod \"e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:04.796316 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.796288 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-istio-envoy\") pod \"e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:04.796495 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.796468 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-credential-socket\") pod \"e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:04.796530 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.796517 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c22d0b-c50f-4f98-a101-846557ad04f6-istio-token\") pod \"e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:04.796581 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.796564 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c22d0b-c50f-4f98-a101-846557ad04f6-istio-podinfo\") pod \"e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:04.832880 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.832828 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0a30f61a-openshift-default-dockercfg-ltp6r\"" Nov 03 20:59:04.833729 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.833698 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m"] Nov 03 20:59:04.897127 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.897041 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45bx4\" (UniqueName: \"kubernetes.io/projected/17c22d0b-c50f-4f98-a101-846557ad04f6-kube-api-access-45bx4\") pod \"e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:04.897127 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.897097 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-istio-envoy\") pod \"e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:04.897127 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.897135 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-credential-socket\") pod \"e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:04.897494 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.897151 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c22d0b-c50f-4f98-a101-846557ad04f6-istio-token\") pod \"e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:04.897494 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.897176 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c22d0b-c50f-4f98-a101-846557ad04f6-istio-podinfo\") pod \"e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:04.897494 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.897223 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-workload-certs\") pod \"e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:04.897494 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.897242 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-istio-data\") pod \"e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:04.897494 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.897267 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c22d0b-c50f-4f98-a101-846557ad04f6-istiod-ca-cert\") pod \"e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:04.897494 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.897298 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-workload-socket\") pod \"e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:04.897805 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.897633 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-credential-socket\") pod \"e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:04.897805 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.897680 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-workload-certs\") pod \"e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:04.897805 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.897734 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-istio-data\") pod \"e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:04.897939 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.897802 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-workload-socket\") pod \"e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:04.898056 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.898034 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c22d0b-c50f-4f98-a101-846557ad04f6-istiod-ca-cert\") pod \"e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:04.899971 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.899946 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-istio-envoy\") pod \"e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:04.900252 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.900231 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c22d0b-c50f-4f98-a101-846557ad04f6-istio-podinfo\") pod \"e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:04.955041 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.955001 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45bx4\" (UniqueName: \"kubernetes.io/projected/17c22d0b-c50f-4f98-a101-846557ad04f6-kube-api-access-45bx4\") pod \"e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:04.955041 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:04.955040 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c22d0b-c50f-4f98-a101-846557ad04f6-istio-token\") pod \"e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:05.099645 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:05.098479 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:05.154758 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:05.154379 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc"] Nov 03 20:59:05.164688 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:05.164642 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:59:05.164795 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:05.164733 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:59:05.164795 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:05.164780 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:59:05.370446 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:05.370330 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" event={"ID":"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a","Type":"ContainerStarted","Data":"cf7cfda68fd242eed2efb9d883bbd42823fe21df4020c6887e791d92c0534616"} Nov 03 20:59:05.370446 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:05.370381 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" event={"ID":"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a","Type":"ContainerStarted","Data":"0192ed7dea206c19139fb30205c1c10b82a036c13c82099e73b700d3cb478e08"} Nov 03 20:59:05.473054 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:05.473019 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m"] Nov 03 20:59:05.476914 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:59:05.476885 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c22d0b_c50f_4f98_a101_846557ad04f6.slice/crio-f71669828f0e4a04c0b11e0c3acdce88c53a738bde14933da30d193220094da0 WatchSource:0}: Error finding container f71669828f0e4a04c0b11e0c3acdce88c53a738bde14933da30d193220094da0: Status 404 returned error can't find the container with id f71669828f0e4a04c0b11e0c3acdce88c53a738bde14933da30d193220094da0 Nov 03 20:59:05.479277 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:05.479244 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:59:05.479361 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:05.479315 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:59:05.479361 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:05.479345 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:59:05.490783 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:05.490689 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" podStartSLOduration=1.490670278 podStartE2EDuration="1.490670278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:05.490140846 +0000 UTC m=+639.957919566" watchObservedRunningTime="2026-11-03 20:59:05.490670278 +0000 UTC m=+639.958448993" Nov 03 20:59:05.737435 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:05.737399 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:06.377145 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:06.377102 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" event={"ID":"17c22d0b-c50f-4f98-a101-846557ad04f6","Type":"ContainerStarted","Data":"aa2dfdca869918753924194a03c1f93651c90072babbc368871343cfa287d4c5"} Nov 03 20:59:06.377345 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:06.377171 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" event={"ID":"17c22d0b-c50f-4f98-a101-846557ad04f6","Type":"ContainerStarted","Data":"f71669828f0e4a04c0b11e0c3acdce88c53a738bde14933da30d193220094da0"} Nov 03 20:59:06.492388 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:06.492319 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" podStartSLOduration=2.492297839 podStartE2EDuration="2.492297839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:06.490982971 +0000 UTC m=+640.958761712" watchObservedRunningTime="2026-11-03 20:59:06.492297839 +0000 UTC m=+640.960076562" Nov 03 20:59:06.745260 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:06.745232 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:07.098896 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:07.098805 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:07.103866 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:07.103837 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:07.383346 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:07.383257 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:07.383346 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:07.383338 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:07.385324 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:07.385297 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:07.385491 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:07.385384 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:10.619315 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:59:10.619273 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 20:59:11.220063 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.220028 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-10bfccd7-6998b5c8c9-x852r"] Nov 03 20:59:11.223683 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.223656 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-10bfccd7-6998b5c8c9-x852r" Nov 03 20:59:11.269807 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.269768 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95462c02-0404-47b2-bd2f-1fd27c989c20-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-10bfccd7-6998b5c8c9-x852r\" (UID: \"95462c02-0404-47b2-bd2f-1fd27c989c20\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-10bfccd7-6998b5c8c9-x852r" Nov 03 20:59:11.269807 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.269808 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcckb\" (UniqueName: \"kubernetes.io/projected/95462c02-0404-47b2-bd2f-1fd27c989c20-kube-api-access-pcckb\") pod \"payload-pre-processing-e2e-ait-10bfccd7-6998b5c8c9-x852r\" (UID: \"95462c02-0404-47b2-bd2f-1fd27c989c20\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-10bfccd7-6998b5c8c9-x852r" Nov 03 20:59:11.322707 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.322664 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-10bfccd7-dockercfg-dcgjn\"" Nov 03 20:59:11.322707 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.322664 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-10bfccd7\"" Nov 03 20:59:11.324459 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.324430 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-10bfccd7-6998b5c8c9-x852r"] Nov 03 20:59:11.368703 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.368667 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-10bfccd7-97649b7f8-66c64"] Nov 03 20:59:11.371418 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.371395 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95462c02-0404-47b2-bd2f-1fd27c989c20-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-10bfccd7-6998b5c8c9-x852r\" (UID: \"95462c02-0404-47b2-bd2f-1fd27c989c20\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-10bfccd7-6998b5c8c9-x852r" Nov 03 20:59:11.371529 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.371426 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcckb\" (UniqueName: \"kubernetes.io/projected/95462c02-0404-47b2-bd2f-1fd27c989c20-kube-api-access-pcckb\") pod \"payload-pre-processing-e2e-ait-10bfccd7-6998b5c8c9-x852r\" (UID: \"95462c02-0404-47b2-bd2f-1fd27c989c20\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-10bfccd7-6998b5c8c9-x852r" Nov 03 20:59:11.372116 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.372093 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95462c02-0404-47b2-bd2f-1fd27c989c20-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-10bfccd7-6998b5c8c9-x852r\" (UID: \"95462c02-0404-47b2-bd2f-1fd27c989c20\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-10bfccd7-6998b5c8c9-x852r" Nov 03 20:59:11.372458 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.372436 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-10bfccd7-97649b7f8-66c64" Nov 03 20:59:11.415730 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.415693 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc"] Nov 03 20:59:11.419523 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.419501 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:11.428598 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.428573 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-10bfccd7-97649b7f8-66c64"] Nov 03 20:59:11.434825 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.434800 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcckb\" (UniqueName: \"kubernetes.io/projected/95462c02-0404-47b2-bd2f-1fd27c989c20-kube-api-access-pcckb\") pod \"payload-pre-processing-e2e-ait-10bfccd7-6998b5c8c9-x852r\" (UID: \"95462c02-0404-47b2-bd2f-1fd27c989c20\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-10bfccd7-6998b5c8c9-x852r" Nov 03 20:59:11.469545 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.469500 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-c9f7f443-openshift-default-dockercfg-pmc5x\"" Nov 03 20:59:11.472311 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.472224 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-workload-socket\") pod \"e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:11.472311 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.472274 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istio-envoy\") pod \"e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:11.472578 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.472334 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-credential-socket\") pod \"e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:11.472578 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.472363 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istiod-ca-cert\") pod \"e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:11.472578 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.472392 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-workload-certs\") pod \"e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:11.472578 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.472417 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88e07263-3ac7-4918-9b6f-234d0a5af683-plugins-config-volume\") pod \"payload-processing-e2e-ait-10bfccd7-97649b7f8-66c64\" (UID: \"88e07263-3ac7-4918-9b6f-234d0a5af683\") " pod="openshift-ingress/payload-processing-e2e-ait-10bfccd7-97649b7f8-66c64" Nov 03 20:59:11.472578 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.472453 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istio-data\") pod \"e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:11.472578 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.472478 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istio-podinfo\") pod \"e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:11.472578 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.472508 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtxkm\" (UniqueName: \"kubernetes.io/projected/1b8f7b11-df18-4a7b-876a-a1ad15c10882-kube-api-access-qtxkm\") pod \"e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:11.472578 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.472539 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh94b\" (UniqueName: \"kubernetes.io/projected/88e07263-3ac7-4918-9b6f-234d0a5af683-kube-api-access-wh94b\") pod \"payload-processing-e2e-ait-10bfccd7-97649b7f8-66c64\" (UID: \"88e07263-3ac7-4918-9b6f-234d0a5af683\") " pod="openshift-ingress/payload-processing-e2e-ait-10bfccd7-97649b7f8-66c64" Nov 03 20:59:11.473089 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.472615 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istio-token\") pod \"e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:11.520025 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.519993 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc"] Nov 03 20:59:11.537899 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.537868 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-10bfccd7-6998b5c8c9-x852r" Nov 03 20:59:11.574442 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.573506 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istio-token\") pod \"e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:11.574442 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.573664 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-workload-socket\") pod \"e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:11.574442 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.573704 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istio-envoy\") pod \"e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:11.574442 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.573802 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-credential-socket\") pod \"e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:11.574442 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.573840 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istiod-ca-cert\") pod \"e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:11.574442 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.573864 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-workload-certs\") pod \"e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:11.574442 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.573888 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88e07263-3ac7-4918-9b6f-234d0a5af683-plugins-config-volume\") pod \"payload-processing-e2e-ait-10bfccd7-97649b7f8-66c64\" (UID: \"88e07263-3ac7-4918-9b6f-234d0a5af683\") " pod="openshift-ingress/payload-processing-e2e-ait-10bfccd7-97649b7f8-66c64" Nov 03 20:59:11.574442 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.573932 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istio-data\") pod \"e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:11.574442 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.573956 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istio-podinfo\") pod \"e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:11.574442 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.573988 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtxkm\" (UniqueName: \"kubernetes.io/projected/1b8f7b11-df18-4a7b-876a-a1ad15c10882-kube-api-access-qtxkm\") pod \"e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:11.574442 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.574022 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh94b\" (UniqueName: \"kubernetes.io/projected/88e07263-3ac7-4918-9b6f-234d0a5af683-kube-api-access-wh94b\") pod \"payload-processing-e2e-ait-10bfccd7-97649b7f8-66c64\" (UID: \"88e07263-3ac7-4918-9b6f-234d0a5af683\") " pod="openshift-ingress/payload-processing-e2e-ait-10bfccd7-97649b7f8-66c64" Nov 03 20:59:11.578110 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.575927 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-workload-socket\") pod \"e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:11.578110 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.576006 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-workload-certs\") pod \"e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:11.578110 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.576714 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88e07263-3ac7-4918-9b6f-234d0a5af683-plugins-config-volume\") pod \"payload-processing-e2e-ait-10bfccd7-97649b7f8-66c64\" (UID: \"88e07263-3ac7-4918-9b6f-234d0a5af683\") " pod="openshift-ingress/payload-processing-e2e-ait-10bfccd7-97649b7f8-66c64" Nov 03 20:59:11.578110 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.577209 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istio-data\") pod \"e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:11.578110 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.577462 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-credential-socket\") pod \"e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:11.578110 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.577978 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istiod-ca-cert\") pod \"e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:11.584130 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.582678 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istio-envoy\") pod \"e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:11.584739 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.584712 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istio-podinfo\") pod \"e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:11.632270 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.632239 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh94b\" (UniqueName: \"kubernetes.io/projected/88e07263-3ac7-4918-9b6f-234d0a5af683-kube-api-access-wh94b\") pod \"payload-processing-e2e-ait-10bfccd7-97649b7f8-66c64\" (UID: \"88e07263-3ac7-4918-9b6f-234d0a5af683\") " pod="openshift-ingress/payload-processing-e2e-ait-10bfccd7-97649b7f8-66c64" Nov 03 20:59:11.632606 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.632568 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istio-token\") pod \"e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:11.637321 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.637288 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtxkm\" (UniqueName: \"kubernetes.io/projected/1b8f7b11-df18-4a7b-876a-a1ad15c10882-kube-api-access-qtxkm\") pod \"e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:11.686467 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.686431 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-10bfccd7-97649b7f8-66c64" Nov 03 20:59:11.733272 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.733236 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:11.920994 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:11.920969 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-10bfccd7-6998b5c8c9-x852r"] Nov 03 20:59:11.923030 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:59:11.923004 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95462c02_0404_47b2_bd2f_1fd27c989c20.slice/crio-9f192e959129dd7c596fb06d795b73b922dc626483fe6e9770ea08cd0700f272 WatchSource:0}: Error finding container 9f192e959129dd7c596fb06d795b73b922dc626483fe6e9770ea08cd0700f272: Status 404 returned error can't find the container with id 9f192e959129dd7c596fb06d795b73b922dc626483fe6e9770ea08cd0700f272 Nov 03 20:59:12.063984 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:12.063954 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-10bfccd7-97649b7f8-66c64"] Nov 03 20:59:12.066641 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:59:12.066611 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e07263_3ac7_4918_9b6f_234d0a5af683.slice/crio-b91f8790508bc97d8ab002116fcca3136eac651f41d7e679ce9c0a72b045c2c8 WatchSource:0}: Error finding container b91f8790508bc97d8ab002116fcca3136eac651f41d7e679ce9c0a72b045c2c8: Status 404 returned error can't find the container with id b91f8790508bc97d8ab002116fcca3136eac651f41d7e679ce9c0a72b045c2c8 Nov 03 20:59:12.113987 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:12.113965 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc"] Nov 03 20:59:12.125184 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:59:12.125151 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b8f7b11_df18_4a7b_876a_a1ad15c10882.slice/crio-bb897bdf499ffc5de3519992b0c2bf2197c9a04265d4a5c0bdd69da4973a8160 WatchSource:0}: Error finding container bb897bdf499ffc5de3519992b0c2bf2197c9a04265d4a5c0bdd69da4973a8160: Status 404 returned error can't find the container with id bb897bdf499ffc5de3519992b0c2bf2197c9a04265d4a5c0bdd69da4973a8160 Nov 03 20:59:12.127604 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:12.127572 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:59:12.127671 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:12.127641 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:59:12.127706 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:12.127673 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:59:12.410036 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:12.409932 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-10bfccd7-6998b5c8c9-x852r" event={"ID":"95462c02-0404-47b2-bd2f-1fd27c989c20","Type":"ContainerStarted","Data":"7e2ac422ad03848a80c894646ee9f674443af6032e2038a4d124828548d316ac"} Nov 03 20:59:12.410036 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:12.410000 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-10bfccd7-6998b5c8c9-x852r" event={"ID":"95462c02-0404-47b2-bd2f-1fd27c989c20","Type":"ContainerStarted","Data":"9f192e959129dd7c596fb06d795b73b922dc626483fe6e9770ea08cd0700f272"} Nov 03 20:59:12.410282 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:12.410053 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-10bfccd7-6998b5c8c9-x852r" Nov 03 20:59:12.411587 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:12.411557 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-10bfccd7-97649b7f8-66c64" event={"ID":"88e07263-3ac7-4918-9b6f-234d0a5af683","Type":"ContainerStarted","Data":"ba6ea1660b010cf3b645c91f939fd765fce35a97ef7e60edad005bccd37cd41a"} Nov 03 20:59:12.411726 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:12.411607 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-10bfccd7-97649b7f8-66c64" event={"ID":"88e07263-3ac7-4918-9b6f-234d0a5af683","Type":"ContainerStarted","Data":"b91f8790508bc97d8ab002116fcca3136eac651f41d7e679ce9c0a72b045c2c8"} Nov 03 20:59:12.411726 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:12.411668 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-10bfccd7-97649b7f8-66c64" Nov 03 20:59:12.413224 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:12.413196 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" event={"ID":"1b8f7b11-df18-4a7b-876a-a1ad15c10882","Type":"ContainerStarted","Data":"c1264547ba64c2072b9204fabb4331a36d7ae9910ffed931f62d8358d162832f"} Nov 03 20:59:12.413322 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:12.413233 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" event={"ID":"1b8f7b11-df18-4a7b-876a-a1ad15c10882","Type":"ContainerStarted","Data":"bb897bdf499ffc5de3519992b0c2bf2197c9a04265d4a5c0bdd69da4973a8160"} Nov 03 20:59:12.520480 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:12.520423 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-10bfccd7-6998b5c8c9-x852r" podStartSLOduration=1.520408355 podStartE2EDuration="1.520408355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:12.519348628 +0000 UTC m=+646.987127352" watchObservedRunningTime="2026-11-03 20:59:12.520408355 +0000 UTC m=+646.988187079" Nov 03 20:59:12.629264 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:12.629209 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-10bfccd7-97649b7f8-66c64" podStartSLOduration=1.629193943 podStartE2EDuration="1.629193943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:12.628415425 +0000 UTC m=+647.096194148" watchObservedRunningTime="2026-11-03 20:59:12.629193943 +0000 UTC m=+647.096972666" Nov 03 20:59:12.734102 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:12.734046 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:12.743872 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:12.743802 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" podStartSLOduration=1.7437860029999999 podStartE2EDuration="1.743786003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:12.742706998 +0000 UTC m=+647.210485722" watchObservedRunningTime="2026-11-03 20:59:12.743786003 +0000 UTC m=+647.211564727" Nov 03 20:59:13.453612 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:13.453577 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk"] Nov 03 20:59:13.457760 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:13.457739 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk" Nov 03 20:59:13.494207 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:13.494170 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/049be06c-1bfb-4b43-a7c7-3230718dd428-metrics-certs\") pod \"maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk\" (UID: \"049be06c-1bfb-4b43-a7c7-3230718dd428\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk" Nov 03 20:59:13.494435 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:13.494401 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/049be06c-1bfb-4b43-a7c7-3230718dd428-openshift-service-ca\") pod \"maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk\" (UID: \"049be06c-1bfb-4b43-a7c7-3230718dd428\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk" Nov 03 20:59:13.494602 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:13.494579 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/049be06c-1bfb-4b43-a7c7-3230718dd428-maas-api-tls\") pod \"maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk\" (UID: \"049be06c-1bfb-4b43-a7c7-3230718dd428\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk" Nov 03 20:59:13.494758 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:13.494740 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl7z4\" (UniqueName: \"kubernetes.io/projected/049be06c-1bfb-4b43-a7c7-3230718dd428-kube-api-access-fl7z4\") pod \"maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk\" (UID: \"049be06c-1bfb-4b43-a7c7-3230718dd428\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk" Nov 03 20:59:13.510206 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:13.510180 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-367d7739-serving-cert\"" Nov 03 20:59:13.524682 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:13.524647 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk"] Nov 03 20:59:13.595540 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:13.595496 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl7z4\" (UniqueName: \"kubernetes.io/projected/049be06c-1bfb-4b43-a7c7-3230718dd428-kube-api-access-fl7z4\") pod \"maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk\" (UID: \"049be06c-1bfb-4b43-a7c7-3230718dd428\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk" Nov 03 20:59:13.595732 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:13.595600 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/049be06c-1bfb-4b43-a7c7-3230718dd428-metrics-certs\") pod \"maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk\" (UID: \"049be06c-1bfb-4b43-a7c7-3230718dd428\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk" Nov 03 20:59:13.595732 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:13.595636 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/049be06c-1bfb-4b43-a7c7-3230718dd428-openshift-service-ca\") pod \"maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk\" (UID: \"049be06c-1bfb-4b43-a7c7-3230718dd428\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk" Nov 03 20:59:13.595732 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:13.595655 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/049be06c-1bfb-4b43-a7c7-3230718dd428-maas-api-tls\") pod \"maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk\" (UID: \"049be06c-1bfb-4b43-a7c7-3230718dd428\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk" Nov 03 20:59:13.595852 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:59:13.595764 5615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-serving-cert: secret "maas-api-e2e-migrate-367d7739-serving-cert" not found Nov 03 20:59:13.595852 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:59:13.595822 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/049be06c-1bfb-4b43-a7c7-3230718dd428-maas-api-tls podName:049be06c-1bfb-4b43-a7c7-3230718dd428 nodeName:}" failed. No retries permitted until 2026-11-03 20:59:14.095805085 +0000 UTC m=+648.563583806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/049be06c-1bfb-4b43-a7c7-3230718dd428-maas-api-tls") pod "maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk" (UID: "049be06c-1bfb-4b43-a7c7-3230718dd428") : secret "maas-api-e2e-migrate-367d7739-serving-cert" not found Nov 03 20:59:13.596375 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:13.596352 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/049be06c-1bfb-4b43-a7c7-3230718dd428-openshift-service-ca\") pod \"maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk\" (UID: \"049be06c-1bfb-4b43-a7c7-3230718dd428\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk" Nov 03 20:59:13.598165 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:13.598145 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/049be06c-1bfb-4b43-a7c7-3230718dd428-metrics-certs\") pod \"maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk\" (UID: \"049be06c-1bfb-4b43-a7c7-3230718dd428\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk" Nov 03 20:59:13.654484 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:13.654445 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl7z4\" (UniqueName: \"kubernetes.io/projected/049be06c-1bfb-4b43-a7c7-3230718dd428-kube-api-access-fl7z4\") pod \"maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk\" (UID: \"049be06c-1bfb-4b43-a7c7-3230718dd428\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk" Nov 03 20:59:13.740797 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:13.740755 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:14.107557 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:14.107469 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/049be06c-1bfb-4b43-a7c7-3230718dd428-maas-api-tls\") pod \"maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk\" (UID: \"049be06c-1bfb-4b43-a7c7-3230718dd428\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk" Nov 03 20:59:14.110055 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:14.110029 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/049be06c-1bfb-4b43-a7c7-3230718dd428-maas-api-tls\") pod \"maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk\" (UID: \"049be06c-1bfb-4b43-a7c7-3230718dd428\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk" Nov 03 20:59:14.371202 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:14.371098 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk" Nov 03 20:59:14.426793 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:14.425198 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:14.426793 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:14.426278 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 20:59:14.738540 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:14.738518 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk"] Nov 03 20:59:14.740693 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:59:14.740660 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049be06c_1bfb_4b43_a7c7_3230718dd428.slice/crio-82bd15a92fa0adf48dbd0eb63fdf0b6070c924fdbfbd8af8fecca5805d4dd184 WatchSource:0}: Error finding container 82bd15a92fa0adf48dbd0eb63fdf0b6070c924fdbfbd8af8fecca5805d4dd184: Status 404 returned error can't find the container with id 82bd15a92fa0adf48dbd0eb63fdf0b6070c924fdbfbd8af8fecca5805d4dd184 Nov 03 20:59:15.431455 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:15.431411 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk" event={"ID":"049be06c-1bfb-4b43-a7c7-3230718dd428","Type":"ContainerStarted","Data":"82bd15a92fa0adf48dbd0eb63fdf0b6070c924fdbfbd8af8fecca5805d4dd184"} Nov 03 20:59:16.282447 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:16.282415 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0a30f61a-787df9f798-rw57q"] Nov 03 20:59:16.286223 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:16.286201 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0a30f61a-787df9f798-rw57q" Nov 03 20:59:16.340339 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:16.340299 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0a30f61a-787df9f798-rw57q"] Nov 03 20:59:16.343709 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:16.343673 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1052f74-edf3-434d-9136-848161742ab4-plugins-config-volume\") pod \"payload-processing-e2e-ait-0a30f61a-787df9f798-rw57q\" (UID: \"b1052f74-edf3-434d-9136-848161742ab4\") " pod="openshift-ingress/payload-processing-e2e-ait-0a30f61a-787df9f798-rw57q" Nov 03 20:59:16.343864 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:16.343763 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdfvg\" (UniqueName: \"kubernetes.io/projected/b1052f74-edf3-434d-9136-848161742ab4-kube-api-access-jdfvg\") pod \"payload-processing-e2e-ait-0a30f61a-787df9f798-rw57q\" (UID: \"b1052f74-edf3-434d-9136-848161742ab4\") " pod="openshift-ingress/payload-processing-e2e-ait-0a30f61a-787df9f798-rw57q" Nov 03 20:59:16.385620 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:16.385578 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0a30f61a\"" Nov 03 20:59:16.388201 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:16.388174 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0a30f61a-dockercfg-x69v2\"" Nov 03 20:59:16.438289 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:16.438254 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk" event={"ID":"049be06c-1bfb-4b43-a7c7-3230718dd428","Type":"ContainerStarted","Data":"5738acec851113cf92433f1bbc88e458e77c9ca2080061c40893c400aae990eb"} Nov 03 20:59:16.438688 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:16.438346 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk" Nov 03 20:59:16.445269 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:16.445232 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1052f74-edf3-434d-9136-848161742ab4-plugins-config-volume\") pod \"payload-processing-e2e-ait-0a30f61a-787df9f798-rw57q\" (UID: \"b1052f74-edf3-434d-9136-848161742ab4\") " pod="openshift-ingress/payload-processing-e2e-ait-0a30f61a-787df9f798-rw57q" Nov 03 20:59:16.445415 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:16.445303 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdfvg\" (UniqueName: \"kubernetes.io/projected/b1052f74-edf3-434d-9136-848161742ab4-kube-api-access-jdfvg\") pod \"payload-processing-e2e-ait-0a30f61a-787df9f798-rw57q\" (UID: \"b1052f74-edf3-434d-9136-848161742ab4\") " pod="openshift-ingress/payload-processing-e2e-ait-0a30f61a-787df9f798-rw57q" Nov 03 20:59:16.445895 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:16.445873 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1052f74-edf3-434d-9136-848161742ab4-plugins-config-volume\") pod \"payload-processing-e2e-ait-0a30f61a-787df9f798-rw57q\" (UID: \"b1052f74-edf3-434d-9136-848161742ab4\") " pod="openshift-ingress/payload-processing-e2e-ait-0a30f61a-787df9f798-rw57q" Nov 03 20:59:16.503576 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:16.503534 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdfvg\" (UniqueName: \"kubernetes.io/projected/b1052f74-edf3-434d-9136-848161742ab4-kube-api-access-jdfvg\") pod \"payload-processing-e2e-ait-0a30f61a-787df9f798-rw57q\" (UID: \"b1052f74-edf3-434d-9136-848161742ab4\") " pod="openshift-ingress/payload-processing-e2e-ait-0a30f61a-787df9f798-rw57q" Nov 03 20:59:16.599170 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:16.599060 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0a30f61a-787df9f798-rw57q" Nov 03 20:59:16.965372 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:16.964922 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk" podStartSLOduration=2.98098536 podStartE2EDuration="3.964900093s" totalImagesPullingTime="983.914733ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:16.557494285 +0000 UTC m=+651.025273009" watchObservedRunningTime="2026-11-03 20:59:16.964900093 +0000 UTC m=+651.432678817" Nov 03 20:59:16.966139 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:16.966112 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0a30f61a-787df9f798-rw57q"] Nov 03 20:59:16.967706 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:59:16.967677 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1052f74_edf3_434d_9136_848161742ab4.slice/crio-ec7985e5bc35d300d7eb6123989daec65d187f837264b24c0f14323c387bfb3e WatchSource:0}: Error finding container ec7985e5bc35d300d7eb6123989daec65d187f837264b24c0f14323c387bfb3e: Status 404 returned error can't find the container with id ec7985e5bc35d300d7eb6123989daec65d187f837264b24c0f14323c387bfb3e Nov 03 20:59:17.446283 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:17.446247 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0a30f61a-787df9f798-rw57q" event={"ID":"b1052f74-edf3-434d-9136-848161742ab4","Type":"ContainerStarted","Data":"58df56557bc172d115dacef46cb5624ff7dbb10fc298c3ba19932ff79df2919a"} Nov 03 20:59:17.446701 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:17.446329 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0a30f61a-787df9f798-rw57q" Nov 03 20:59:17.446701 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:17.446348 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0a30f61a-787df9f798-rw57q" event={"ID":"b1052f74-edf3-434d-9136-848161742ab4","Type":"ContainerStarted","Data":"ec7985e5bc35d300d7eb6123989daec65d187f837264b24c0f14323c387bfb3e"} Nov 03 20:59:17.557566 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:17.557494 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0a30f61a-787df9f798-rw57q" podStartSLOduration=1.557481315 podStartE2EDuration="1.557481315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:17.556248214 +0000 UTC m=+652.024026938" watchObservedRunningTime="2026-11-03 20:59:17.557481315 +0000 UTC m=+652.025260037" Nov 03 20:59:21.278099 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:21.278041 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5b32d5a5-ffcff7b4c-r5wtm"] Nov 03 20:59:21.281948 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:21.281927 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b32d5a5-ffcff7b4c-r5wtm" Nov 03 20:59:21.382666 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:21.382632 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5b32d5a5\"" Nov 03 20:59:21.382873 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:21.382764 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5b32d5a5-dockercfg-95zhg\"" Nov 03 20:59:21.384614 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:21.384578 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5b32d5a5-ffcff7b4c-r5wtm"] Nov 03 20:59:21.420889 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:21.419778 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c889837-acc2-436c-8ede-741e552b47bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5b32d5a5-ffcff7b4c-r5wtm\" (UID: \"1c889837-acc2-436c-8ede-741e552b47bc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5b32d5a5-ffcff7b4c-r5wtm" Nov 03 20:59:21.420889 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:21.419915 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8wk9\" (UniqueName: \"kubernetes.io/projected/1c889837-acc2-436c-8ede-741e552b47bc-kube-api-access-z8wk9\") pod \"payload-pre-processing-e2e-ait-5b32d5a5-ffcff7b4c-r5wtm\" (UID: \"1c889837-acc2-436c-8ede-741e552b47bc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5b32d5a5-ffcff7b4c-r5wtm" Nov 03 20:59:21.521373 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:21.521336 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8wk9\" (UniqueName: \"kubernetes.io/projected/1c889837-acc2-436c-8ede-741e552b47bc-kube-api-access-z8wk9\") pod \"payload-pre-processing-e2e-ait-5b32d5a5-ffcff7b4c-r5wtm\" (UID: \"1c889837-acc2-436c-8ede-741e552b47bc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5b32d5a5-ffcff7b4c-r5wtm" Nov 03 20:59:21.521544 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:21.521442 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c889837-acc2-436c-8ede-741e552b47bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5b32d5a5-ffcff7b4c-r5wtm\" (UID: \"1c889837-acc2-436c-8ede-741e552b47bc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5b32d5a5-ffcff7b4c-r5wtm" Nov 03 20:59:21.522206 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:21.522187 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c889837-acc2-436c-8ede-741e552b47bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5b32d5a5-ffcff7b4c-r5wtm\" (UID: \"1c889837-acc2-436c-8ede-741e552b47bc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5b32d5a5-ffcff7b4c-r5wtm" Nov 03 20:59:21.580283 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:21.580215 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8wk9\" (UniqueName: \"kubernetes.io/projected/1c889837-acc2-436c-8ede-741e552b47bc-kube-api-access-z8wk9\") pod \"payload-pre-processing-e2e-ait-5b32d5a5-ffcff7b4c-r5wtm\" (UID: \"1c889837-acc2-436c-8ede-741e552b47bc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5b32d5a5-ffcff7b4c-r5wtm" Nov 03 20:59:21.594864 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:21.594836 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b32d5a5-ffcff7b4c-r5wtm" Nov 03 20:59:21.961577 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:21.961546 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5b32d5a5-ffcff7b4c-r5wtm"] Nov 03 20:59:21.963726 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:59:21.963700 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c889837_acc2_436c_8ede_741e552b47bc.slice/crio-b0fd5c29e95e914f1d64bc59f353aba17a41e858239c86f56542ebfac20958d3 WatchSource:0}: Error finding container b0fd5c29e95e914f1d64bc59f353aba17a41e858239c86f56542ebfac20958d3: Status 404 returned error can't find the container with id b0fd5c29e95e914f1d64bc59f353aba17a41e858239c86f56542ebfac20958d3 Nov 03 20:59:22.452064 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:22.452038 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk" Nov 03 20:59:22.474982 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:22.474943 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b32d5a5-ffcff7b4c-r5wtm" event={"ID":"1c889837-acc2-436c-8ede-741e552b47bc","Type":"ContainerStarted","Data":"c7bef82dafc6221b381bdaf0237079eb60953c19386273736536a45c0ccf3e5c"} Nov 03 20:59:22.475175 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:22.475000 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b32d5a5-ffcff7b4c-r5wtm" event={"ID":"1c889837-acc2-436c-8ede-741e552b47bc","Type":"ContainerStarted","Data":"b0fd5c29e95e914f1d64bc59f353aba17a41e858239c86f56542ebfac20958d3"} Nov 03 20:59:22.475175 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:22.475027 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b32d5a5-ffcff7b4c-r5wtm" Nov 03 20:59:22.677794 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:22.677737 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b32d5a5-ffcff7b4c-r5wtm" podStartSLOduration=1.677722401 podStartE2EDuration="1.677722401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:22.676394052 +0000 UTC m=+657.144172775" watchObservedRunningTime="2026-11-03 20:59:22.677722401 +0000 UTC m=+657.145501124" Nov 03 20:59:23.361794 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:23.361761 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc"] Nov 03 20:59:23.365816 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:23.365790 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc" Nov 03 20:59:23.421838 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:23.421807 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-10bfccd7-97649b7f8-66c64" Nov 03 20:59:23.422336 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:23.422311 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-10bfccd7-6998b5c8c9-x852r" Nov 03 20:59:23.449558 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:23.449516 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6093506-8a41-4f1e-ae12-09e9346f6dc5-maas-api-tls\") pod \"maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc\" (UID: \"a6093506-8a41-4f1e-ae12-09e9346f6dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc" Nov 03 20:59:23.449558 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:23.449561 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6093506-8a41-4f1e-ae12-09e9346f6dc5-metrics-certs\") pod \"maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc\" (UID: \"a6093506-8a41-4f1e-ae12-09e9346f6dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc" Nov 03 20:59:23.449841 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:23.449587 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6093506-8a41-4f1e-ae12-09e9346f6dc5-openshift-service-ca\") pod \"maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc\" (UID: \"a6093506-8a41-4f1e-ae12-09e9346f6dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc" Nov 03 20:59:23.449841 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:23.449728 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9k7k\" (UniqueName: \"kubernetes.io/projected/a6093506-8a41-4f1e-ae12-09e9346f6dc5-kube-api-access-g9k7k\") pod \"maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc\" (UID: \"a6093506-8a41-4f1e-ae12-09e9346f6dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc" Nov 03 20:59:23.465676 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:23.465646 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5b32d5a5-serving-cert\"" Nov 03 20:59:23.467385 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:23.467366 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc"] Nov 03 20:59:23.550340 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:23.550304 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9k7k\" (UniqueName: \"kubernetes.io/projected/a6093506-8a41-4f1e-ae12-09e9346f6dc5-kube-api-access-g9k7k\") pod \"maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc\" (UID: \"a6093506-8a41-4f1e-ae12-09e9346f6dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc" Nov 03 20:59:23.550544 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:23.550460 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6093506-8a41-4f1e-ae12-09e9346f6dc5-maas-api-tls\") pod \"maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc\" (UID: \"a6093506-8a41-4f1e-ae12-09e9346f6dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc" Nov 03 20:59:23.550614 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:59:23.550543 5615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-serving-cert: secret "maas-api-e2e-ait-5b32d5a5-serving-cert" not found Nov 03 20:59:23.550614 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:59:23.550607 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6093506-8a41-4f1e-ae12-09e9346f6dc5-maas-api-tls podName:a6093506-8a41-4f1e-ae12-09e9346f6dc5 nodeName:}" failed. No retries permitted until 2026-11-03 20:59:24.050593315 +0000 UTC m=+658.518372016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a6093506-8a41-4f1e-ae12-09e9346f6dc5-maas-api-tls") pod "maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc" (UID: "a6093506-8a41-4f1e-ae12-09e9346f6dc5") : secret "maas-api-e2e-ait-5b32d5a5-serving-cert" not found Nov 03 20:59:23.550746 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:23.550651 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6093506-8a41-4f1e-ae12-09e9346f6dc5-metrics-certs\") pod \"maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc\" (UID: \"a6093506-8a41-4f1e-ae12-09e9346f6dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc" Nov 03 20:59:23.550746 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:23.550710 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6093506-8a41-4f1e-ae12-09e9346f6dc5-openshift-service-ca\") pod \"maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc\" (UID: \"a6093506-8a41-4f1e-ae12-09e9346f6dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc" Nov 03 20:59:23.551378 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:23.551354 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6093506-8a41-4f1e-ae12-09e9346f6dc5-openshift-service-ca\") pod \"maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc\" (UID: \"a6093506-8a41-4f1e-ae12-09e9346f6dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc" Nov 03 20:59:23.553278 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:23.553258 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6093506-8a41-4f1e-ae12-09e9346f6dc5-metrics-certs\") pod \"maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc\" (UID: \"a6093506-8a41-4f1e-ae12-09e9346f6dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc" Nov 03 20:59:23.614368 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:23.614293 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9k7k\" (UniqueName: \"kubernetes.io/projected/a6093506-8a41-4f1e-ae12-09e9346f6dc5-kube-api-access-g9k7k\") pod \"maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc\" (UID: \"a6093506-8a41-4f1e-ae12-09e9346f6dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc" Nov 03 20:59:24.056857 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:24.056802 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6093506-8a41-4f1e-ae12-09e9346f6dc5-maas-api-tls\") pod \"maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc\" (UID: \"a6093506-8a41-4f1e-ae12-09e9346f6dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc" Nov 03 20:59:24.059368 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:24.059346 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6093506-8a41-4f1e-ae12-09e9346f6dc5-maas-api-tls\") pod \"maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc\" (UID: \"a6093506-8a41-4f1e-ae12-09e9346f6dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc" Nov 03 20:59:24.279019 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:24.278974 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc" Nov 03 20:59:24.692773 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:24.692750 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc"] Nov 03 20:59:24.694679 ip-10-0-129-192 kubenswrapper[5615]: W1103 20:59:24.694652 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6093506_8a41_4f1e_ae12_09e9346f6dc5.slice/crio-ab23910daa25fb3a7672cf9183417bb23f34bc88a98178b128a61b695beed683 WatchSource:0}: Error finding container ab23910daa25fb3a7672cf9183417bb23f34bc88a98178b128a61b695beed683: Status 404 returned error can't find the container with id ab23910daa25fb3a7672cf9183417bb23f34bc88a98178b128a61b695beed683 Nov 03 20:59:25.491441 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:25.491405 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc" event={"ID":"a6093506-8a41-4f1e-ae12-09e9346f6dc5","Type":"ContainerStarted","Data":"ab23910daa25fb3a7672cf9183417bb23f34bc88a98178b128a61b695beed683"} Nov 03 20:59:26.497612 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:26.497573 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc" event={"ID":"a6093506-8a41-4f1e-ae12-09e9346f6dc5","Type":"ContainerStarted","Data":"e55342c07a3eec82568223a3a7ab75d188fa02c9ee3eb9bff416fc89d68d7286"} Nov 03 20:59:26.498234 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:26.497717 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc" Nov 03 20:59:26.609499 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:26.609445 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc" podStartSLOduration=2.722535889 podStartE2EDuration="3.609432752s" totalImagesPullingTime="886.896863ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:26.609027056 +0000 UTC m=+661.076805779" watchObservedRunningTime="2026-11-03 20:59:26.609432752 +0000 UTC m=+661.077211469" Nov 03 20:59:28.456156 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:28.456124 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0a30f61a-787df9f798-rw57q" Nov 03 20:59:32.523887 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:32.523852 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc" Nov 03 20:59:33.483414 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:33.483381 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b32d5a5-ffcff7b4c-r5wtm" Nov 03 20:59:37.388328 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:37.387161 5615 patch_prober.go:29] interesting pod/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 20:59:37.388328 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:37.387246 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" podUID="17c22d0b-c50f-4f98-a101-846557ad04f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 20:59:37.570882 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:37.570805 5615 generic.go:356] "Generic (PLEG): container finished" podID="17c22d0b-c50f-4f98-a101-846557ad04f6" containerID="aa2dfdca869918753924194a03c1f93651c90072babbc368871343cfa287d4c5" exitCode=0 Nov 03 20:59:37.570882 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:37.570850 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" event={"ID":"17c22d0b-c50f-4f98-a101-846557ad04f6","Type":"ContainerDied","Data":"aa2dfdca869918753924194a03c1f93651c90072babbc368871343cfa287d4c5"} Nov 03 20:59:37.571252 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:37.571237 5615 scope.go:122] "RemoveContainer" containerID="aa2dfdca869918753924194a03c1f93651c90072babbc368871343cfa287d4c5" Nov 03 20:59:37.572502 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:37.572473 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:59:37.572568 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:37.572541 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:59:37.572710 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:37.572604 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:59:38.578808 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:38.578771 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" event={"ID":"17c22d0b-c50f-4f98-a101-846557ad04f6","Type":"ContainerStarted","Data":"0029526f0180075ad6e9355b1dd3cdc634a661625d9c23f98c751b95b558c2ab"} Nov 03 20:59:39.107788 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:39.107753 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:39.108218 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:39.108183 5615 patch_prober.go:29] interesting pod/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 20:59:39.108895 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:39.108861 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" podUID="17c22d0b-c50f-4f98-a101-846557ad04f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 20:59:40.102545 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:40.102504 5615 patch_prober.go:29] interesting pod/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 20:59:40.103203 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:40.103171 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" podUID="17c22d0b-c50f-4f98-a101-846557ad04f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 20:59:40.641891 ip-10-0-129-192 kubenswrapper[5615]: E1103 20:59:40.641855 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 20:59:41.102107 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:41.101475 5615 patch_prober.go:29] interesting pod/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 20:59:41.102107 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:41.101554 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" podUID="17c22d0b-c50f-4f98-a101-846557ad04f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 20:59:42.116798 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:42.116746 5615 patch_prober.go:29] interesting pod/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 20:59:42.117324 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:42.116835 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" podUID="17c22d0b-c50f-4f98-a101-846557ad04f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 20:59:43.122753 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:43.122714 5615 patch_prober.go:29] interesting pod/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 20:59:43.123359 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:43.123325 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" podUID="17c22d0b-c50f-4f98-a101-846557ad04f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 20:59:44.103166 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:44.103129 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:44.103396 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:44.103377 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:44.104132 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:44.104117 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 20:59:44.634269 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:44.634232 5615 generic.go:356] "Generic (PLEG): container finished" podID="919ca8bf-b699-4bb0-88e1-7b71b2d5b11a" containerID="cf7cfda68fd242eed2efb9d883bbd42823fe21df4020c6887e791d92c0534616" exitCode=0 Nov 03 20:59:44.634793 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:44.634309 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" event={"ID":"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a","Type":"ContainerDied","Data":"cf7cfda68fd242eed2efb9d883bbd42823fe21df4020c6887e791d92c0534616"} Nov 03 20:59:44.634911 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:44.634807 5615 scope.go:122] "RemoveContainer" containerID="cf7cfda68fd242eed2efb9d883bbd42823fe21df4020c6887e791d92c0534616" Nov 03 20:59:44.636031 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:44.636001 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:59:44.636124 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:44.636086 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:59:44.636170 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:44.636123 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:59:45.642739 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:45.642692 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" event={"ID":"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a","Type":"ContainerStarted","Data":"92c48ef302fee4f2027d54218d65e45f4656f8fc13e76846a353432ad5c722e4"} Nov 03 20:59:45.733535 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:45.733490 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:45.733807 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:45.733784 5615 patch_prober.go:29] interesting pod/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 20:59:45.733892 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:45.733840 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" podUID="919ca8bf-b699-4bb0-88e1-7b71b2d5b11a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 20:59:46.733708 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:46.733666 5615 patch_prober.go:29] interesting pod/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 20:59:46.734152 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:46.733725 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" podUID="919ca8bf-b699-4bb0-88e1-7b71b2d5b11a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 20:59:47.734009 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:47.733968 5615 patch_prober.go:29] interesting pod/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 20:59:47.734538 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:47.734046 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" podUID="919ca8bf-b699-4bb0-88e1-7b71b2d5b11a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 20:59:48.734229 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:48.734188 5615 patch_prober.go:29] interesting pod/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 20:59:48.734635 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:48.734257 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" podUID="919ca8bf-b699-4bb0-88e1-7b71b2d5b11a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 20:59:49.734100 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:49.734038 5615 patch_prober.go:29] interesting pod/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 20:59:49.734287 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:49.734122 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" podUID="919ca8bf-b699-4bb0-88e1-7b71b2d5b11a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 20:59:50.747777 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:50.747740 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:50.748564 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:50.748537 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 20:59:50.749276 ip-10-0-129-192 kubenswrapper[5615]: I1103 20:59:50.749245 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 21:00:00.169452 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:00.169415 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-tpxf2"] Nov 03 21:00:00.173153 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:00.173122 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-tpxf2" Nov 03 21:00:00.222621 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:00.222588 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\"" Nov 03 21:00:00.264907 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:00.264662 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895660-tpxf2\" (UID: \"2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-tpxf2" Nov 03 21:00:00.264907 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:00.264751 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck7r7\" (UniqueName: \"kubernetes.io/projected/2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8-kube-api-access-ck7r7\") pod \"maas-api-key-cleanup-29895660-tpxf2\" (UID: \"2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-tpxf2" Nov 03 21:00:00.366305 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:00.366263 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck7r7\" (UniqueName: \"kubernetes.io/projected/2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8-kube-api-access-ck7r7\") pod \"maas-api-key-cleanup-29895660-tpxf2\" (UID: \"2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-tpxf2" Nov 03 21:00:00.366512 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:00.366370 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895660-tpxf2\" (UID: \"2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-tpxf2" Nov 03 21:00:00.366949 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:00.366925 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895660-tpxf2\" (UID: \"2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-tpxf2" Nov 03 21:00:00.379006 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:00.378973 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-tpxf2"] Nov 03 21:00:00.425099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:00.425008 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck7r7\" (UniqueName: \"kubernetes.io/projected/2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8-kube-api-access-ck7r7\") pod \"maas-api-key-cleanup-29895660-tpxf2\" (UID: \"2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-tpxf2" Nov 03 21:00:00.466331 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:00.466281 5615 patch_prober.go:29] interesting pod/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:00:00.466486 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:00.466376 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" podUID="1b8f7b11-df18-4a7b-876a-a1ad15c10882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:00.490839 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:00.490798 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-tpxf2" Nov 03 21:00:00.861554 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:00.861500 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-tpxf2"] Nov 03 21:00:00.868132 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:00:00.868096 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fbb7cc2_ef9c_4f5b_93cc_0faf069109d8.slice/crio-7189ee64996ff2d3bef702dae3d11ec7095ea2747146f1183485e5324a4970b6 WatchSource:0}: Error finding container 7189ee64996ff2d3bef702dae3d11ec7095ea2747146f1183485e5324a4970b6: Status 404 returned error can't find the container with id 7189ee64996ff2d3bef702dae3d11ec7095ea2747146f1183485e5324a4970b6 Nov 03 21:00:01.755685 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:01.755647 5615 generic.go:356] "Generic (PLEG): container finished" podID="1b8f7b11-df18-4a7b-876a-a1ad15c10882" containerID="c1264547ba64c2072b9204fabb4331a36d7ae9910ffed931f62d8358d162832f" exitCode=0 Nov 03 21:00:01.756311 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:01.755726 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" event={"ID":"1b8f7b11-df18-4a7b-876a-a1ad15c10882","Type":"ContainerDied","Data":"c1264547ba64c2072b9204fabb4331a36d7ae9910ffed931f62d8358d162832f"} Nov 03 21:00:01.756311 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:01.756240 5615 scope.go:122] "RemoveContainer" containerID="c1264547ba64c2072b9204fabb4331a36d7ae9910ffed931f62d8358d162832f" Nov 03 21:00:01.757111 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:01.757040 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-tpxf2" event={"ID":"2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8","Type":"ContainerStarted","Data":"7189ee64996ff2d3bef702dae3d11ec7095ea2747146f1183485e5324a4970b6"} Nov 03 21:00:01.757543 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:01.757507 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:00:01.757599 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:01.757586 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:00:01.757634 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:01.757619 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:00:02.764290 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:02.764251 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" event={"ID":"1b8f7b11-df18-4a7b-876a-a1ad15c10882","Type":"ContainerStarted","Data":"28c0713135daac31f9a02fba48a531f81486c0be7590574ccff5659ab65b902e"} Nov 03 21:00:03.734559 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:03.734520 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 21:00:03.734847 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:03.734574 5615 patch_prober.go:29] interesting pod/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 03 21:00:03.734986 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:03.734882 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" podUID="1b8f7b11-df18-4a7b-876a-a1ad15c10882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 03 21:00:04.734255 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:04.734215 5615 patch_prober.go:29] interesting pod/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 03 21:00:04.734753 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:04.734284 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" podUID="1b8f7b11-df18-4a7b-876a-a1ad15c10882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 03 21:00:04.896468 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:04.896432 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj"] Nov 03 21:00:04.923995 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:04.923954 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj" Nov 03 21:00:04.948968 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:04.948922 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3075b40b-4848-4e09-9d7e-a1b2eac692b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj\" (UID: \"3075b40b-4848-4e09-9d7e-a1b2eac692b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj" Nov 03 21:00:04.949150 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:04.949095 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3075b40b-4848-4e09-9d7e-a1b2eac692b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj\" (UID: \"3075b40b-4848-4e09-9d7e-a1b2eac692b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj" Nov 03 21:00:04.949220 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:04.949153 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv2hz\" (UniqueName: \"kubernetes.io/projected/3075b40b-4848-4e09-9d7e-a1b2eac692b1-kube-api-access-xv2hz\") pod \"maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj\" (UID: \"3075b40b-4848-4e09-9d7e-a1b2eac692b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj" Nov 03 21:00:04.953351 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:04.953306 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj"] Nov 03 21:00:05.022169 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:05.021299 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:00:05.049925 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:05.049895 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3075b40b-4848-4e09-9d7e-a1b2eac692b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj\" (UID: \"3075b40b-4848-4e09-9d7e-a1b2eac692b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj" Nov 03 21:00:05.050115 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:05.049993 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3075b40b-4848-4e09-9d7e-a1b2eac692b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj\" (UID: \"3075b40b-4848-4e09-9d7e-a1b2eac692b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj" Nov 03 21:00:05.050115 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:05.050026 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv2hz\" (UniqueName: \"kubernetes.io/projected/3075b40b-4848-4e09-9d7e-a1b2eac692b1-kube-api-access-xv2hz\") pod \"maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj\" (UID: \"3075b40b-4848-4e09-9d7e-a1b2eac692b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj" Nov 03 21:00:05.050551 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:05.050528 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3075b40b-4848-4e09-9d7e-a1b2eac692b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj\" (UID: \"3075b40b-4848-4e09-9d7e-a1b2eac692b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj" Nov 03 21:00:05.052354 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:05.052336 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3075b40b-4848-4e09-9d7e-a1b2eac692b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj\" (UID: \"3075b40b-4848-4e09-9d7e-a1b2eac692b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj" Nov 03 21:00:05.124852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:05.124796 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv2hz\" (UniqueName: \"kubernetes.io/projected/3075b40b-4848-4e09-9d7e-a1b2eac692b1-kube-api-access-xv2hz\") pod \"maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj\" (UID: \"3075b40b-4848-4e09-9d7e-a1b2eac692b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj" Nov 03 21:00:05.237493 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:05.237456 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj" Nov 03 21:00:05.623644 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:05.623469 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj"] Nov 03 21:00:05.626119 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:00:05.626070 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3075b40b_4848_4e09_9d7e_a1b2eac692b1.slice/crio-3c5435dc5f37d56caddfbe9ff6971b53b8ac6b9b2a109014c065167f5af04f0c WatchSource:0}: Error finding container 3c5435dc5f37d56caddfbe9ff6971b53b8ac6b9b2a109014c065167f5af04f0c: Status 404 returned error can't find the container with id 3c5435dc5f37d56caddfbe9ff6971b53b8ac6b9b2a109014c065167f5af04f0c Nov 03 21:00:05.628574 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:05.628554 5615 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:00:05.734718 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:05.734681 5615 patch_prober.go:29] interesting pod/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 03 21:00:05.735152 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:05.734735 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" podUID="1b8f7b11-df18-4a7b-876a-a1ad15c10882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 03 21:00:05.788138 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:05.788048 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj" event={"ID":"3075b40b-4848-4e09-9d7e-a1b2eac692b1","Type":"ContainerStarted","Data":"3c5435dc5f37d56caddfbe9ff6971b53b8ac6b9b2a109014c065167f5af04f0c"} Nov 03 21:00:06.590191 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:06.590152 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0a30f61a-787df9f798-rw57q"] Nov 03 21:00:06.734760 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:06.734720 5615 patch_prober.go:29] interesting pod/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 03 21:00:06.735243 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:06.734806 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" podUID="1b8f7b11-df18-4a7b-876a-a1ad15c10882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 03 21:00:07.734589 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:07.734555 5615 patch_prober.go:29] interesting pod/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 03 21:00:07.734776 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:07.734628 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" podUID="1b8f7b11-df18-4a7b-876a-a1ad15c10882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 03 21:00:08.739528 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:08.739498 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 21:00:08.739987 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:08.739778 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 21:00:08.740531 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:08.740509 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 21:00:08.804772 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:08.804742 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-tpxf2" event={"ID":"2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8","Type":"ContainerStarted","Data":"70180a5a5846772da15ee42fcc4251e000a15ccb9031b12fa9a141c989b26134"} Nov 03 21:00:09.092432 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:09.092385 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-tpxf2" podStartSLOduration=1.630180614 podStartE2EDuration="9.09236998s" totalImagesPullingTime="7.462189366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:09.092133942 +0000 UTC m=+703.559912667" watchObservedRunningTime="2026-11-03 21:00:09.09236998 +0000 UTC m=+703.560148704" Nov 03 21:00:09.756485 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:09.756436 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk"] Nov 03 21:00:09.757672 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:09.757363 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk" podUID="049be06c-1bfb-4b43-a7c7-3230718dd428" containerName="maas-api" containerID="cri-o://5738acec851113cf92433f1bbc88e458e77c9ca2080061c40893c400aae990eb" gracePeriod=30 Nov 03 21:00:09.815394 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:09.815363 5615 generic.go:356] "Generic (PLEG): container finished" podID="2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8" containerID="70180a5a5846772da15ee42fcc4251e000a15ccb9031b12fa9a141c989b26134" exitCode=60 Nov 03 21:00:09.815591 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:09.815456 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-tpxf2" event={"ID":"2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8","Type":"ContainerDied","Data":"70180a5a5846772da15ee42fcc4251e000a15ccb9031b12fa9a141c989b26134"} Nov 03 21:00:09.815717 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:09.815698 5615 scope.go:122] "RemoveContainer" containerID="70180a5a5846772da15ee42fcc4251e000a15ccb9031b12fa9a141c989b26134" Nov 03 21:00:10.078242 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:10.078216 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk" Nov 03 21:00:10.132384 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:10.131424 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/049be06c-1bfb-4b43-a7c7-3230718dd428-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/049be06c-1bfb-4b43-a7c7-3230718dd428-maas-api-tls\") pod \"049be06c-1bfb-4b43-a7c7-3230718dd428\" (UID: \"049be06c-1bfb-4b43-a7c7-3230718dd428\") " Nov 03 21:00:10.132384 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:10.131497 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/049be06c-1bfb-4b43-a7c7-3230718dd428-metrics-certs\" (UniqueName: \"kubernetes.io/secret/049be06c-1bfb-4b43-a7c7-3230718dd428-metrics-certs\") pod \"049be06c-1bfb-4b43-a7c7-3230718dd428\" (UID: \"049be06c-1bfb-4b43-a7c7-3230718dd428\") " Nov 03 21:00:10.132384 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:10.131568 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/049be06c-1bfb-4b43-a7c7-3230718dd428-kube-api-access-fl7z4\" (UniqueName: \"kubernetes.io/projected/049be06c-1bfb-4b43-a7c7-3230718dd428-kube-api-access-fl7z4\") pod \"049be06c-1bfb-4b43-a7c7-3230718dd428\" (UID: \"049be06c-1bfb-4b43-a7c7-3230718dd428\") " Nov 03 21:00:10.132384 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:10.131692 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/049be06c-1bfb-4b43-a7c7-3230718dd428-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/049be06c-1bfb-4b43-a7c7-3230718dd428-openshift-service-ca\") pod \"049be06c-1bfb-4b43-a7c7-3230718dd428\" (UID: \"049be06c-1bfb-4b43-a7c7-3230718dd428\") " Nov 03 21:00:10.135451 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:10.135410 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/049be06c-1bfb-4b43-a7c7-3230718dd428-maas-api-tls" pod "049be06c-1bfb-4b43-a7c7-3230718dd428" (UID: "049be06c-1bfb-4b43-a7c7-3230718dd428"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:00:10.136214 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:10.136170 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049be06c-1bfb-4b43-a7c7-3230718dd428-kube-api-access-fl7z4" pod "049be06c-1bfb-4b43-a7c7-3230718dd428" (UID: "049be06c-1bfb-4b43-a7c7-3230718dd428"). InnerVolumeSpecName "kube-api-access-fl7z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:10.137935 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:10.137910 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/049be06c-1bfb-4b43-a7c7-3230718dd428-metrics-certs" pod "049be06c-1bfb-4b43-a7c7-3230718dd428" (UID: "049be06c-1bfb-4b43-a7c7-3230718dd428"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:00:10.140181 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:10.140131 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/049be06c-1bfb-4b43-a7c7-3230718dd428-openshift-service-ca" pod "049be06c-1bfb-4b43-a7c7-3230718dd428" (UID: "049be06c-1bfb-4b43-a7c7-3230718dd428"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:10.233623 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:10.233591 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/049be06c-1bfb-4b43-a7c7-3230718dd428-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:10.233765 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:10.233632 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/049be06c-1bfb-4b43-a7c7-3230718dd428-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:10.233765 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:10.233649 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl7z4\" (UniqueName: \"kubernetes.io/projected/049be06c-1bfb-4b43-a7c7-3230718dd428-kube-api-access-fl7z4\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:10.233765 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:10.233665 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/049be06c-1bfb-4b43-a7c7-3230718dd428-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:10.620543 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:00:10.620448 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:00:10.824596 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:10.824556 5615 generic.go:356] "Generic (PLEG): container finished" podID="2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8" containerID="871ca39471cec057d72c788a70c295aa0c44a7197287af84183a1f928fc49cab" exitCode=60 Nov 03 21:00:10.825096 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:10.824631 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-tpxf2" event={"ID":"2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8","Type":"ContainerDied","Data":"871ca39471cec057d72c788a70c295aa0c44a7197287af84183a1f928fc49cab"} Nov 03 21:00:10.825096 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:10.824686 5615 scope.go:122] "RemoveContainer" containerID="70180a5a5846772da15ee42fcc4251e000a15ccb9031b12fa9a141c989b26134" Nov 03 21:00:10.825096 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:10.825004 5615 scope.go:122] "RemoveContainer" containerID="871ca39471cec057d72c788a70c295aa0c44a7197287af84183a1f928fc49cab" Nov 03 21:00:10.825359 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:00:10.825334 5615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895660-tpxf2_odh-ai-gateway-infra(2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-tpxf2" podUID="2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8" Nov 03 21:00:10.826264 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:10.826241 5615 generic.go:356] "Generic (PLEG): container finished" podID="049be06c-1bfb-4b43-a7c7-3230718dd428" containerID="5738acec851113cf92433f1bbc88e458e77c9ca2080061c40893c400aae990eb" exitCode=0 Nov 03 21:00:10.826379 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:10.826298 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk" Nov 03 21:00:10.826379 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:10.826323 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk" event={"ID":"049be06c-1bfb-4b43-a7c7-3230718dd428","Type":"ContainerDied","Data":"5738acec851113cf92433f1bbc88e458e77c9ca2080061c40893c400aae990eb"} Nov 03 21:00:10.826494 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:10.826385 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk" event={"ID":"049be06c-1bfb-4b43-a7c7-3230718dd428","Type":"ContainerDied","Data":"82bd15a92fa0adf48dbd0eb63fdf0b6070c924fdbfbd8af8fecca5805d4dd184"} Nov 03 21:00:10.828047 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:10.828023 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj" event={"ID":"3075b40b-4848-4e09-9d7e-a1b2eac692b1","Type":"ContainerStarted","Data":"6e1ca4e19799be5f61b4f5b34235bc0f16b44b723b31723ece34ba3481ef8f88"} Nov 03 21:00:10.856040 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:10.856015 5615 scope.go:122] "RemoveContainer" containerID="5738acec851113cf92433f1bbc88e458e77c9ca2080061c40893c400aae990eb" Nov 03 21:00:10.865279 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:10.865253 5615 scope.go:122] "RemoveContainer" containerID="5738acec851113cf92433f1bbc88e458e77c9ca2080061c40893c400aae990eb" Nov 03 21:00:10.865600 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:00:10.865578 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5738acec851113cf92433f1bbc88e458e77c9ca2080061c40893c400aae990eb\": container with ID starting with 5738acec851113cf92433f1bbc88e458e77c9ca2080061c40893c400aae990eb not found: ID does not exist" containerID="5738acec851113cf92433f1bbc88e458e77c9ca2080061c40893c400aae990eb" Nov 03 21:00:10.865662 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:10.865614 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5738acec851113cf92433f1bbc88e458e77c9ca2080061c40893c400aae990eb"} err="failed to get container status \"5738acec851113cf92433f1bbc88e458e77c9ca2080061c40893c400aae990eb\": rpc error: code = NotFound desc = could not find container \"5738acec851113cf92433f1bbc88e458e77c9ca2080061c40893c400aae990eb\": container with ID starting with 5738acec851113cf92433f1bbc88e458e77c9ca2080061c40893c400aae990eb not found: ID does not exist" Nov 03 21:00:11.040523 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:11.040490 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk"] Nov 03 21:00:11.048999 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:11.048969 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-367d7739-7684b8f45b-x9rsk"] Nov 03 21:00:11.157676 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:11.157617 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj" podStartSLOduration=2.4228719659999998 podStartE2EDuration="7.157603901s" totalImagesPullingTime="4.734731935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:11.15681001 +0000 UTC m=+705.624588733" watchObservedRunningTime="2026-11-03 21:00:11.157603901 +0000 UTC m=+705.625382624" Nov 03 21:00:11.591337 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:11.591281 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0a30f61a-787df9f798-rw57q" podUID="b1052f74-edf3-434d-9136-848161742ab4" containerName="payload-processing" containerID="cri-o://58df56557bc172d115dacef46cb5624ff7dbb10fc298c3ba19932ff79df2919a" gracePeriod=25 Nov 03 21:00:11.650464 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:11.650385 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="049be06c-1bfb-4b43-a7c7-3230718dd428" path="/var/lib/kubelet/pods/049be06c-1bfb-4b43-a7c7-3230718dd428/volumes" Nov 03 21:00:11.839229 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:11.839198 5615 generic.go:356] "Generic (PLEG): container finished" podID="b1052f74-edf3-434d-9136-848161742ab4" containerID="58df56557bc172d115dacef46cb5624ff7dbb10fc298c3ba19932ff79df2919a" exitCode=0 Nov 03 21:00:11.839820 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:11.839323 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0a30f61a-787df9f798-rw57q" event={"ID":"b1052f74-edf3-434d-9136-848161742ab4","Type":"ContainerDied","Data":"58df56557bc172d115dacef46cb5624ff7dbb10fc298c3ba19932ff79df2919a"} Nov 03 21:00:11.841223 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:11.841195 5615 generic.go:356] "Generic (PLEG): container finished" podID="3075b40b-4848-4e09-9d7e-a1b2eac692b1" containerID="6e1ca4e19799be5f61b4f5b34235bc0f16b44b723b31723ece34ba3481ef8f88" exitCode=0 Nov 03 21:00:11.841342 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:11.841274 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj" event={"ID":"3075b40b-4848-4e09-9d7e-a1b2eac692b1","Type":"ContainerDied","Data":"6e1ca4e19799be5f61b4f5b34235bc0f16b44b723b31723ece34ba3481ef8f88"} Nov 03 21:00:11.843223 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:11.843204 5615 scope.go:122] "RemoveContainer" containerID="871ca39471cec057d72c788a70c295aa0c44a7197287af84183a1f928fc49cab" Nov 03 21:00:11.843493 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:00:11.843470 5615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895660-tpxf2_odh-ai-gateway-infra(2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-tpxf2" podUID="2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8" Nov 03 21:00:11.870750 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:11.870725 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0a30f61a-787df9f798-rw57q" Nov 03 21:00:11.950715 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:11.950630 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1052f74-edf3-434d-9136-848161742ab4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1052f74-edf3-434d-9136-848161742ab4-plugins-config-volume\") pod \"b1052f74-edf3-434d-9136-848161742ab4\" (UID: \"b1052f74-edf3-434d-9136-848161742ab4\") " Nov 03 21:00:11.950715 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:11.950686 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1052f74-edf3-434d-9136-848161742ab4-kube-api-access-jdfvg\" (UniqueName: \"kubernetes.io/projected/b1052f74-edf3-434d-9136-848161742ab4-kube-api-access-jdfvg\") pod \"b1052f74-edf3-434d-9136-848161742ab4\" (UID: \"b1052f74-edf3-434d-9136-848161742ab4\") " Nov 03 21:00:11.951037 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:11.951010 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1052f74-edf3-434d-9136-848161742ab4-plugins-config-volume" pod "b1052f74-edf3-434d-9136-848161742ab4" (UID: "b1052f74-edf3-434d-9136-848161742ab4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:11.951206 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:11.951186 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1052f74-edf3-434d-9136-848161742ab4-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:11.953595 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:11.953568 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1052f74-edf3-434d-9136-848161742ab4-kube-api-access-jdfvg" pod "b1052f74-edf3-434d-9136-848161742ab4" (UID: "b1052f74-edf3-434d-9136-848161742ab4"). InnerVolumeSpecName "kube-api-access-jdfvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:12.053977 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:12.051908 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdfvg\" (UniqueName: \"kubernetes.io/projected/b1052f74-edf3-434d-9136-848161742ab4-kube-api-access-jdfvg\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:12.849853 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:12.849822 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0a30f61a-787df9f798-rw57q" Nov 03 21:00:12.850538 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:12.849863 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0a30f61a-787df9f798-rw57q" event={"ID":"b1052f74-edf3-434d-9136-848161742ab4","Type":"ContainerDied","Data":"ec7985e5bc35d300d7eb6123989daec65d187f837264b24c0f14323c387bfb3e"} Nov 03 21:00:12.850538 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:12.849928 5615 scope.go:122] "RemoveContainer" containerID="58df56557bc172d115dacef46cb5624ff7dbb10fc298c3ba19932ff79df2919a" Nov 03 21:00:12.996609 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:12.996583 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj" Nov 03 21:00:13.010578 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:13.010540 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0a30f61a-787df9f798-rw57q"] Nov 03 21:00:13.058233 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:13.058206 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0a30f61a-787df9f798-rw57q"] Nov 03 21:00:13.060584 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:13.060555 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3075b40b-4848-4e09-9d7e-a1b2eac692b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3075b40b-4848-4e09-9d7e-a1b2eac692b1-maas-api-openshift-ca\") pod \"3075b40b-4848-4e09-9d7e-a1b2eac692b1\" (UID: \"3075b40b-4848-4e09-9d7e-a1b2eac692b1\") " Nov 03 21:00:13.060740 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:13.060716 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3075b40b-4848-4e09-9d7e-a1b2eac692b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3075b40b-4848-4e09-9d7e-a1b2eac692b1-maas-api-xks-ca\") pod \"3075b40b-4848-4e09-9d7e-a1b2eac692b1\" (UID: \"3075b40b-4848-4e09-9d7e-a1b2eac692b1\") " Nov 03 21:00:13.060844 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:13.060829 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3075b40b-4848-4e09-9d7e-a1b2eac692b1-kube-api-access-xv2hz\" (UniqueName: \"kubernetes.io/projected/3075b40b-4848-4e09-9d7e-a1b2eac692b1-kube-api-access-xv2hz\") pod \"3075b40b-4848-4e09-9d7e-a1b2eac692b1\" (UID: \"3075b40b-4848-4e09-9d7e-a1b2eac692b1\") " Nov 03 21:00:13.060948 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:13.060928 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3075b40b-4848-4e09-9d7e-a1b2eac692b1-maas-api-openshift-ca" pod "3075b40b-4848-4e09-9d7e-a1b2eac692b1" (UID: "3075b40b-4848-4e09-9d7e-a1b2eac692b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:13.061184 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:13.061165 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3075b40b-4848-4e09-9d7e-a1b2eac692b1-maas-api-openshift-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:13.062357 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:13.062336 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m"] Nov 03 21:00:13.062612 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:13.062581 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" podUID="17c22d0b-c50f-4f98-a101-846557ad04f6" containerName="istio-proxy" containerID="cri-o://0029526f0180075ad6e9355b1dd3cdc634a661625d9c23f98c751b95b558c2ab" gracePeriod=30 Nov 03 21:00:13.064623 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:13.064586 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3075b40b-4848-4e09-9d7e-a1b2eac692b1-maas-api-xks-ca" pod "3075b40b-4848-4e09-9d7e-a1b2eac692b1" (UID: "3075b40b-4848-4e09-9d7e-a1b2eac692b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:00:13.065147 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:13.065119 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3075b40b-4848-4e09-9d7e-a1b2eac692b1-kube-api-access-xv2hz" pod "3075b40b-4848-4e09-9d7e-a1b2eac692b1" (UID: "3075b40b-4848-4e09-9d7e-a1b2eac692b1"). InnerVolumeSpecName "kube-api-access-xv2hz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:13.166298 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:13.166210 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv2hz\" (UniqueName: \"kubernetes.io/projected/3075b40b-4848-4e09-9d7e-a1b2eac692b1-kube-api-access-xv2hz\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:13.166298 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:13.166251 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3075b40b-4848-4e09-9d7e-a1b2eac692b1-maas-api-xks-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:13.651170 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:13.651138 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1052f74-edf3-434d-9136-848161742ab4" path="/var/lib/kubelet/pods/b1052f74-edf3-434d-9136-848161742ab4/volumes" Nov 03 21:00:13.857964 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:13.857935 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj" Nov 03 21:00:13.858400 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:13.857935 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj" event={"ID":"3075b40b-4848-4e09-9d7e-a1b2eac692b1","Type":"ContainerDied","Data":"3c5435dc5f37d56caddfbe9ff6971b53b8ac6b9b2a109014c065167f5af04f0c"} Nov 03 21:00:13.858400 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:13.858069 5615 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c5435dc5f37d56caddfbe9ff6971b53b8ac6b9b2a109014c065167f5af04f0c" Nov 03 21:00:14.098625 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.098586 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5b32d5a5-ffcff7b4c-r5wtm"] Nov 03 21:00:14.106165 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.106128 5615 patch_prober.go:29] interesting pod/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:00:14.106322 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.106183 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" podUID="17c22d0b-c50f-4f98-a101-846557ad04f6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:00:14.768009 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.767979 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd"] Nov 03 21:00:14.768485 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.768471 5615 cpu_manager.go:414] "containerMap: removing container" podUID="049be06c-1bfb-4b43-a7c7-3230718dd428" containerName="maas-api" Nov 03 21:00:14.768537 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.768487 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="049be06c-1bfb-4b43-a7c7-3230718dd428" containerName="maas-api" Nov 03 21:00:14.768537 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.768503 5615 cpu_manager.go:414] "containerMap: removing container" podUID="3075b40b-4848-4e09-9d7e-a1b2eac692b1" containerName="revoke-keys" Nov 03 21:00:14.768537 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.768508 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3075b40b-4848-4e09-9d7e-a1b2eac692b1" containerName="revoke-keys" Nov 03 21:00:14.768537 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.768523 5615 cpu_manager.go:414] "containerMap: removing container" podUID="b1052f74-edf3-434d-9136-848161742ab4" containerName="payload-processing" Nov 03 21:00:14.768537 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.768528 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1052f74-edf3-434d-9136-848161742ab4" containerName="payload-processing" Nov 03 21:00:14.768688 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.768590 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="3075b40b-4848-4e09-9d7e-a1b2eac692b1" containerName="revoke-keys" Nov 03 21:00:14.768688 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.768599 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1052f74-edf3-434d-9136-848161742ab4" containerName="payload-processing" Nov 03 21:00:14.768688 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.768609 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="049be06c-1bfb-4b43-a7c7-3230718dd428" containerName="maas-api" Nov 03 21:00:14.771310 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.771286 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:14.820979 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.820949 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-320f60-openshift-default-dockercfg-jct6g\"" Nov 03 21:00:14.829070 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.829033 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd"] Nov 03 21:00:14.885652 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.885616 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istio-envoy\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:14.886050 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.885665 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istio-podinfo\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:14.886050 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.885730 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-workload-certs\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:14.886050 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.885781 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istiod-ca-cert\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:14.886050 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.885902 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istio-token\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:14.886050 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.885938 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-662h7\" (UniqueName: \"kubernetes.io/projected/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-kube-api-access-662h7\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:14.886050 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.885958 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istio-data\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:14.886050 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.886028 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-workload-socket\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:14.886306 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.886108 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-credential-socket\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:14.987504 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.987472 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istio-token\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:14.987683 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.987517 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-662h7\" (UniqueName: \"kubernetes.io/projected/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-kube-api-access-662h7\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:14.987683 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.987622 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istio-data\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:14.987683 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.987649 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-workload-socket\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:14.987833 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.987694 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-credential-socket\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:14.987833 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.987717 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istio-envoy\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:14.987833 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.987751 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istio-podinfo\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:14.987833 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.987777 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-workload-certs\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:14.988030 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.987848 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istiod-ca-cert\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:14.988161 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.988135 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istio-data\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:14.988234 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.988199 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-workload-socket\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:14.988289 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.988224 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-credential-socket\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:14.988351 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.988288 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-workload-certs\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:14.988575 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.988558 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istiod-ca-cert\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:14.990070 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.990051 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istio-envoy\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:14.990423 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:14.990400 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istio-podinfo\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:15.045616 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:15.045551 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istio-token\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:15.045743 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:15.045705 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-662h7\" (UniqueName: \"kubernetes.io/projected/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-kube-api-access-662h7\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:15.088106 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:15.083738 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:15.510354 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:15.510329 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd"] Nov 03 21:00:15.511845 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:00:15.511816 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b3bf8b_24eb_4b93_80a1_fabdf14cff21.slice/crio-142d09c80e8017ef51249ae1a18b613f7a6d99a7410b140104253e23a8635e6d WatchSource:0}: Error finding container 142d09c80e8017ef51249ae1a18b613f7a6d99a7410b140104253e23a8635e6d: Status 404 returned error can't find the container with id 142d09c80e8017ef51249ae1a18b613f7a6d99a7410b140104253e23a8635e6d Nov 03 21:00:15.514545 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:15.514502 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:00:15.514634 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:15.514580 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:00:15.514634 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:15.514610 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:00:15.871607 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:15.871514 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" event={"ID":"11b3bf8b-24eb-4b93-80a1-fabdf14cff21","Type":"ContainerStarted","Data":"5da03baf1f33f94c77a73c9a13a54392fcf186b52f9c17a702b785901e25abd4"} Nov 03 21:00:15.871607 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:15.871563 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" event={"ID":"11b3bf8b-24eb-4b93-80a1-fabdf14cff21","Type":"ContainerStarted","Data":"142d09c80e8017ef51249ae1a18b613f7a6d99a7410b140104253e23a8635e6d"} Nov 03 21:00:15.986568 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:15.986518 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" podStartSLOduration=1.9865046579999999 podStartE2EDuration="1.986504658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:15.985289567 +0000 UTC m=+710.453068289" watchObservedRunningTime="2026-11-03 21:00:15.986504658 +0000 UTC m=+710.454283377" Nov 03 21:00:16.084473 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:16.084443 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:16.090158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:16.090131 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:16.876746 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:16.876717 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:16.877683 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:16.877663 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:17.159463 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:17.159383 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-10bfccd7-97649b7f8-66c64"] Nov 03 21:00:17.209376 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:17.209347 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-10bfccd7-6998b5c8c9-x852r"] Nov 03 21:00:18.327829 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.327764 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 21:00:18.435214 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.435177 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-workload-certs\") pod \"17c22d0b-c50f-4f98-a101-846557ad04f6\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " Nov 03 21:00:18.435411 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.435232 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-credential-socket\") pod \"17c22d0b-c50f-4f98-a101-846557ad04f6\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " Nov 03 21:00:18.435411 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.435267 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17c22d0b-c50f-4f98-a101-846557ad04f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c22d0b-c50f-4f98-a101-846557ad04f6-istio-podinfo\") pod \"17c22d0b-c50f-4f98-a101-846557ad04f6\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " Nov 03 21:00:18.435411 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.435290 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-workload-socket\") pod \"17c22d0b-c50f-4f98-a101-846557ad04f6\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " Nov 03 21:00:18.435411 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.435318 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-istio-envoy\") pod \"17c22d0b-c50f-4f98-a101-846557ad04f6\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " Nov 03 21:00:18.435411 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.435387 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17c22d0b-c50f-4f98-a101-846557ad04f6-kube-api-access-45bx4\" (UniqueName: \"kubernetes.io/projected/17c22d0b-c50f-4f98-a101-846557ad04f6-kube-api-access-45bx4\") pod \"17c22d0b-c50f-4f98-a101-846557ad04f6\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " Nov 03 21:00:18.435679 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.435436 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-istio-data\") pod \"17c22d0b-c50f-4f98-a101-846557ad04f6\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " Nov 03 21:00:18.435679 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.435475 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17c22d0b-c50f-4f98-a101-846557ad04f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c22d0b-c50f-4f98-a101-846557ad04f6-istiod-ca-cert\") pod \"17c22d0b-c50f-4f98-a101-846557ad04f6\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " Nov 03 21:00:18.435679 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.435553 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17c22d0b-c50f-4f98-a101-846557ad04f6-istio-token\" (UniqueName: \"kubernetes.io/projected/17c22d0b-c50f-4f98-a101-846557ad04f6-istio-token\") pod \"17c22d0b-c50f-4f98-a101-846557ad04f6\" (UID: \"17c22d0b-c50f-4f98-a101-846557ad04f6\") " Nov 03 21:00:18.435679 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.435546 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-workload-certs" pod "17c22d0b-c50f-4f98-a101-846557ad04f6" (UID: "17c22d0b-c50f-4f98-a101-846557ad04f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:18.435679 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.435564 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-credential-socket" pod "17c22d0b-c50f-4f98-a101-846557ad04f6" (UID: "17c22d0b-c50f-4f98-a101-846557ad04f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:18.435952 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.435727 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-workload-socket" pod "17c22d0b-c50f-4f98-a101-846557ad04f6" (UID: "17c22d0b-c50f-4f98-a101-846557ad04f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:18.435952 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.435721 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-istio-data" pod "17c22d0b-c50f-4f98-a101-846557ad04f6" (UID: "17c22d0b-c50f-4f98-a101-846557ad04f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:18.435952 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.435927 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c22d0b-c50f-4f98-a101-846557ad04f6-istiod-ca-cert" pod "17c22d0b-c50f-4f98-a101-846557ad04f6" (UID: "17c22d0b-c50f-4f98-a101-846557ad04f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:18.436141 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.436022 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:18.436141 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.436042 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c22d0b-c50f-4f98-a101-846557ad04f6-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:18.436141 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.436057 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:18.436141 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.436093 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:18.436141 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.436108 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:18.437556 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.437522 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17c22d0b-c50f-4f98-a101-846557ad04f6-istio-podinfo" pod "17c22d0b-c50f-4f98-a101-846557ad04f6" (UID: "17c22d0b-c50f-4f98-a101-846557ad04f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:00:18.437782 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.437761 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-istio-envoy" pod "17c22d0b-c50f-4f98-a101-846557ad04f6" (UID: "17c22d0b-c50f-4f98-a101-846557ad04f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:18.437848 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.437766 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c22d0b-c50f-4f98-a101-846557ad04f6-istio-token" pod "17c22d0b-c50f-4f98-a101-846557ad04f6" (UID: "17c22d0b-c50f-4f98-a101-846557ad04f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:18.437848 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.437817 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c22d0b-c50f-4f98-a101-846557ad04f6-kube-api-access-45bx4" pod "17c22d0b-c50f-4f98-a101-846557ad04f6" (UID: "17c22d0b-c50f-4f98-a101-846557ad04f6"). InnerVolumeSpecName "kube-api-access-45bx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:18.537336 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.537298 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c22d0b-c50f-4f98-a101-846557ad04f6-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:18.537336 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.537331 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c22d0b-c50f-4f98-a101-846557ad04f6-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:18.537336 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.537341 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c22d0b-c50f-4f98-a101-846557ad04f6-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:18.537585 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.537350 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45bx4\" (UniqueName: \"kubernetes.io/projected/17c22d0b-c50f-4f98-a101-846557ad04f6-kube-api-access-45bx4\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:18.889875 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.889833 5615 generic.go:356] "Generic (PLEG): container finished" podID="17c22d0b-c50f-4f98-a101-846557ad04f6" containerID="0029526f0180075ad6e9355b1dd3cdc634a661625d9c23f98c751b95b558c2ab" exitCode=0 Nov 03 21:00:18.890103 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.889893 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" Nov 03 21:00:18.890103 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.889928 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" event={"ID":"17c22d0b-c50f-4f98-a101-846557ad04f6","Type":"ContainerDied","Data":"0029526f0180075ad6e9355b1dd3cdc634a661625d9c23f98c751b95b558c2ab"} Nov 03 21:00:18.890103 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.889986 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m" event={"ID":"17c22d0b-c50f-4f98-a101-846557ad04f6","Type":"ContainerDied","Data":"f71669828f0e4a04c0b11e0c3acdce88c53a738bde14933da30d193220094da0"} Nov 03 21:00:18.890103 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.890011 5615 scope.go:122] "RemoveContainer" containerID="0029526f0180075ad6e9355b1dd3cdc634a661625d9c23f98c751b95b558c2ab" Nov 03 21:00:18.900139 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.900117 5615 scope.go:122] "RemoveContainer" containerID="aa2dfdca869918753924194a03c1f93651c90072babbc368871343cfa287d4c5" Nov 03 21:00:18.909403 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.909380 5615 scope.go:122] "RemoveContainer" containerID="0029526f0180075ad6e9355b1dd3cdc634a661625d9c23f98c751b95b558c2ab" Nov 03 21:00:18.909711 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:00:18.909687 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0029526f0180075ad6e9355b1dd3cdc634a661625d9c23f98c751b95b558c2ab\": container with ID starting with 0029526f0180075ad6e9355b1dd3cdc634a661625d9c23f98c751b95b558c2ab not found: ID does not exist" containerID="0029526f0180075ad6e9355b1dd3cdc634a661625d9c23f98c751b95b558c2ab" Nov 03 21:00:18.909795 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.909722 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0029526f0180075ad6e9355b1dd3cdc634a661625d9c23f98c751b95b558c2ab"} err="failed to get container status \"0029526f0180075ad6e9355b1dd3cdc634a661625d9c23f98c751b95b558c2ab\": rpc error: code = NotFound desc = could not find container \"0029526f0180075ad6e9355b1dd3cdc634a661625d9c23f98c751b95b558c2ab\": container with ID starting with 0029526f0180075ad6e9355b1dd3cdc634a661625d9c23f98c751b95b558c2ab not found: ID does not exist" Nov 03 21:00:18.909795 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.909740 5615 scope.go:122] "RemoveContainer" containerID="aa2dfdca869918753924194a03c1f93651c90072babbc368871343cfa287d4c5" Nov 03 21:00:18.910024 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:00:18.910008 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa2dfdca869918753924194a03c1f93651c90072babbc368871343cfa287d4c5\": container with ID starting with aa2dfdca869918753924194a03c1f93651c90072babbc368871343cfa287d4c5 not found: ID does not exist" containerID="aa2dfdca869918753924194a03c1f93651c90072babbc368871343cfa287d4c5" Nov 03 21:00:18.910120 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:18.910036 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa2dfdca869918753924194a03c1f93651c90072babbc368871343cfa287d4c5"} err="failed to get container status \"aa2dfdca869918753924194a03c1f93651c90072babbc368871343cfa287d4c5\": rpc error: code = NotFound desc = could not find container \"aa2dfdca869918753924194a03c1f93651c90072babbc368871343cfa287d4c5\": container with ID starting with aa2dfdca869918753924194a03c1f93651c90072babbc368871343cfa287d4c5 not found: ID does not exist" Nov 03 21:00:19.049972 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:19.049939 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m"] Nov 03 21:00:19.097337 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:19.097302 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0a30f61a-openshift-default-6858b4ddbf-h9t9m"] Nov 03 21:00:19.099629 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:19.099596 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b32d5a5-ffcff7b4c-r5wtm" podUID="1c889837-acc2-436c-8ede-741e552b47bc" containerName="payload-pre-processing" containerID="cri-o://c7bef82dafc6221b381bdaf0237079eb60953c19386273736536a45c0ccf3e5c" gracePeriod=25 Nov 03 21:00:19.379104 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:19.379058 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b32d5a5-ffcff7b4c-r5wtm" Nov 03 21:00:19.454853 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:19.454815 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c889837-acc2-436c-8ede-741e552b47bc-kube-api-access-z8wk9\" (UniqueName: \"kubernetes.io/projected/1c889837-acc2-436c-8ede-741e552b47bc-kube-api-access-z8wk9\") pod \"1c889837-acc2-436c-8ede-741e552b47bc\" (UID: \"1c889837-acc2-436c-8ede-741e552b47bc\") " Nov 03 21:00:19.455572 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:19.454970 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c889837-acc2-436c-8ede-741e552b47bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c889837-acc2-436c-8ede-741e552b47bc-plugins-config-volume\") pod \"1c889837-acc2-436c-8ede-741e552b47bc\" (UID: \"1c889837-acc2-436c-8ede-741e552b47bc\") " Nov 03 21:00:19.455713 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:19.455646 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c889837-acc2-436c-8ede-741e552b47bc-plugins-config-volume" pod "1c889837-acc2-436c-8ede-741e552b47bc" (UID: "1c889837-acc2-436c-8ede-741e552b47bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:19.457351 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:19.457294 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c889837-acc2-436c-8ede-741e552b47bc-kube-api-access-z8wk9" pod "1c889837-acc2-436c-8ede-741e552b47bc" (UID: "1c889837-acc2-436c-8ede-741e552b47bc"). InnerVolumeSpecName "kube-api-access-z8wk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:19.556920 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:19.556882 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8wk9\" (UniqueName: \"kubernetes.io/projected/1c889837-acc2-436c-8ede-741e552b47bc-kube-api-access-z8wk9\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:19.556920 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:19.556911 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c889837-acc2-436c-8ede-741e552b47bc-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:19.651494 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:19.651460 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17c22d0b-c50f-4f98-a101-846557ad04f6" path="/var/lib/kubelet/pods/17c22d0b-c50f-4f98-a101-846557ad04f6/volumes" Nov 03 21:00:19.895844 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:19.895810 5615 generic.go:356] "Generic (PLEG): container finished" podID="1c889837-acc2-436c-8ede-741e552b47bc" containerID="c7bef82dafc6221b381bdaf0237079eb60953c19386273736536a45c0ccf3e5c" exitCode=0 Nov 03 21:00:19.896006 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:19.895856 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b32d5a5-ffcff7b4c-r5wtm" Nov 03 21:00:19.896006 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:19.895896 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b32d5a5-ffcff7b4c-r5wtm" event={"ID":"1c889837-acc2-436c-8ede-741e552b47bc","Type":"ContainerDied","Data":"c7bef82dafc6221b381bdaf0237079eb60953c19386273736536a45c0ccf3e5c"} Nov 03 21:00:19.896006 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:19.895946 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b32d5a5-ffcff7b4c-r5wtm" event={"ID":"1c889837-acc2-436c-8ede-741e552b47bc","Type":"ContainerDied","Data":"b0fd5c29e95e914f1d64bc59f353aba17a41e858239c86f56542ebfac20958d3"} Nov 03 21:00:19.896006 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:19.895969 5615 scope.go:122] "RemoveContainer" containerID="c7bef82dafc6221b381bdaf0237079eb60953c19386273736536a45c0ccf3e5c" Nov 03 21:00:19.904891 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:19.904873 5615 scope.go:122] "RemoveContainer" containerID="c7bef82dafc6221b381bdaf0237079eb60953c19386273736536a45c0ccf3e5c" Nov 03 21:00:19.905187 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:00:19.905167 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7bef82dafc6221b381bdaf0237079eb60953c19386273736536a45c0ccf3e5c\": container with ID starting with c7bef82dafc6221b381bdaf0237079eb60953c19386273736536a45c0ccf3e5c not found: ID does not exist" containerID="c7bef82dafc6221b381bdaf0237079eb60953c19386273736536a45c0ccf3e5c" Nov 03 21:00:19.905259 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:19.905196 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7bef82dafc6221b381bdaf0237079eb60953c19386273736536a45c0ccf3e5c"} err="failed to get container status \"c7bef82dafc6221b381bdaf0237079eb60953c19386273736536a45c0ccf3e5c\": rpc error: code = NotFound desc = could not find container \"c7bef82dafc6221b381bdaf0237079eb60953c19386273736536a45c0ccf3e5c\": container with ID starting with c7bef82dafc6221b381bdaf0237079eb60953c19386273736536a45c0ccf3e5c not found: ID does not exist" Nov 03 21:00:20.051308 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:20.051271 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5b32d5a5-ffcff7b4c-r5wtm"] Nov 03 21:00:20.098254 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:20.098216 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5b32d5a5-ffcff7b4c-r5wtm"] Nov 03 21:00:21.651065 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:21.651034 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c889837-acc2-436c-8ede-741e552b47bc" path="/var/lib/kubelet/pods/1c889837-acc2-436c-8ede-741e552b47bc/volumes" Nov 03 21:00:22.160432 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.160367 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-10bfccd7-97649b7f8-66c64" podUID="88e07263-3ac7-4918-9b6f-234d0a5af683" containerName="payload-processing" containerID="cri-o://ba6ea1660b010cf3b645c91f939fd765fce35a97ef7e60edad005bccd37cd41a" gracePeriod=25 Nov 03 21:00:22.210333 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.210289 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-10bfccd7-6998b5c8c9-x852r" podUID="95462c02-0404-47b2-bd2f-1fd27c989c20" containerName="payload-pre-processing" containerID="cri-o://7e2ac422ad03848a80c894646ee9f674443af6032e2038a4d124828548d316ac" gracePeriod=25 Nov 03 21:00:22.463955 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.463925 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-10bfccd7-97649b7f8-66c64" Nov 03 21:00:22.495218 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.495189 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-10bfccd7-6998b5c8c9-x852r" Nov 03 21:00:22.590760 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.590729 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e07263-3ac7-4918-9b6f-234d0a5af683-kube-api-access-wh94b\" (UniqueName: \"kubernetes.io/projected/88e07263-3ac7-4918-9b6f-234d0a5af683-kube-api-access-wh94b\") pod \"88e07263-3ac7-4918-9b6f-234d0a5af683\" (UID: \"88e07263-3ac7-4918-9b6f-234d0a5af683\") " Nov 03 21:00:22.590968 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.590767 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95462c02-0404-47b2-bd2f-1fd27c989c20-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95462c02-0404-47b2-bd2f-1fd27c989c20-plugins-config-volume\") pod \"95462c02-0404-47b2-bd2f-1fd27c989c20\" (UID: \"95462c02-0404-47b2-bd2f-1fd27c989c20\") " Nov 03 21:00:22.590968 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.590786 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88e07263-3ac7-4918-9b6f-234d0a5af683-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88e07263-3ac7-4918-9b6f-234d0a5af683-plugins-config-volume\") pod \"88e07263-3ac7-4918-9b6f-234d0a5af683\" (UID: \"88e07263-3ac7-4918-9b6f-234d0a5af683\") " Nov 03 21:00:22.590968 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.590883 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95462c02-0404-47b2-bd2f-1fd27c989c20-kube-api-access-pcckb\" (UniqueName: \"kubernetes.io/projected/95462c02-0404-47b2-bd2f-1fd27c989c20-kube-api-access-pcckb\") pod \"95462c02-0404-47b2-bd2f-1fd27c989c20\" (UID: \"95462c02-0404-47b2-bd2f-1fd27c989c20\") " Nov 03 21:00:22.591706 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.591199 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e07263-3ac7-4918-9b6f-234d0a5af683-plugins-config-volume" pod "88e07263-3ac7-4918-9b6f-234d0a5af683" (UID: "88e07263-3ac7-4918-9b6f-234d0a5af683"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:22.591706 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.591202 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95462c02-0404-47b2-bd2f-1fd27c989c20-plugins-config-volume" pod "95462c02-0404-47b2-bd2f-1fd27c989c20" (UID: "95462c02-0404-47b2-bd2f-1fd27c989c20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:22.593023 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.592994 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e07263-3ac7-4918-9b6f-234d0a5af683-kube-api-access-wh94b" pod "88e07263-3ac7-4918-9b6f-234d0a5af683" (UID: "88e07263-3ac7-4918-9b6f-234d0a5af683"). InnerVolumeSpecName "kube-api-access-wh94b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:22.593130 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.593001 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95462c02-0404-47b2-bd2f-1fd27c989c20-kube-api-access-pcckb" pod "95462c02-0404-47b2-bd2f-1fd27c989c20" (UID: "95462c02-0404-47b2-bd2f-1fd27c989c20"). InnerVolumeSpecName "kube-api-access-pcckb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:22.651026 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.650867 5615 scope.go:122] "RemoveContainer" containerID="871ca39471cec057d72c788a70c295aa0c44a7197287af84183a1f928fc49cab" Nov 03 21:00:22.692651 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.692620 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcckb\" (UniqueName: \"kubernetes.io/projected/95462c02-0404-47b2-bd2f-1fd27c989c20-kube-api-access-pcckb\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:22.692651 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.692650 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh94b\" (UniqueName: \"kubernetes.io/projected/88e07263-3ac7-4918-9b6f-234d0a5af683-kube-api-access-wh94b\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:22.693116 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.692668 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95462c02-0404-47b2-bd2f-1fd27c989c20-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:22.693116 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.692681 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88e07263-3ac7-4918-9b6f-234d0a5af683-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:22.898922 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.893728 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj"] Nov 03 21:00:22.900984 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.900955 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10bfccd7-5cjqj"] Nov 03 21:00:22.914911 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.914884 5615 generic.go:356] "Generic (PLEG): container finished" podID="88e07263-3ac7-4918-9b6f-234d0a5af683" containerID="ba6ea1660b010cf3b645c91f939fd765fce35a97ef7e60edad005bccd37cd41a" exitCode=0 Nov 03 21:00:22.915029 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.914964 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-10bfccd7-97649b7f8-66c64" event={"ID":"88e07263-3ac7-4918-9b6f-234d0a5af683","Type":"ContainerDied","Data":"ba6ea1660b010cf3b645c91f939fd765fce35a97ef7e60edad005bccd37cd41a"} Nov 03 21:00:22.915029 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.914981 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-10bfccd7-97649b7f8-66c64" Nov 03 21:00:22.915164 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.915042 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-10bfccd7-97649b7f8-66c64" event={"ID":"88e07263-3ac7-4918-9b6f-234d0a5af683","Type":"ContainerDied","Data":"b91f8790508bc97d8ab002116fcca3136eac651f41d7e679ce9c0a72b045c2c8"} Nov 03 21:00:22.915164 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.915087 5615 scope.go:122] "RemoveContainer" containerID="ba6ea1660b010cf3b645c91f939fd765fce35a97ef7e60edad005bccd37cd41a" Nov 03 21:00:22.916364 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.916340 5615 generic.go:356] "Generic (PLEG): container finished" podID="95462c02-0404-47b2-bd2f-1fd27c989c20" containerID="7e2ac422ad03848a80c894646ee9f674443af6032e2038a4d124828548d316ac" exitCode=0 Nov 03 21:00:22.916464 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.916399 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-10bfccd7-6998b5c8c9-x852r" Nov 03 21:00:22.916464 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.916419 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-10bfccd7-6998b5c8c9-x852r" event={"ID":"95462c02-0404-47b2-bd2f-1fd27c989c20","Type":"ContainerDied","Data":"7e2ac422ad03848a80c894646ee9f674443af6032e2038a4d124828548d316ac"} Nov 03 21:00:22.916556 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.916466 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-10bfccd7-6998b5c8c9-x852r" event={"ID":"95462c02-0404-47b2-bd2f-1fd27c989c20","Type":"ContainerDied","Data":"9f192e959129dd7c596fb06d795b73b922dc626483fe6e9770ea08cd0700f272"} Nov 03 21:00:22.925524 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.925502 5615 scope.go:122] "RemoveContainer" containerID="ba6ea1660b010cf3b645c91f939fd765fce35a97ef7e60edad005bccd37cd41a" Nov 03 21:00:22.925844 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:00:22.925819 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba6ea1660b010cf3b645c91f939fd765fce35a97ef7e60edad005bccd37cd41a\": container with ID starting with ba6ea1660b010cf3b645c91f939fd765fce35a97ef7e60edad005bccd37cd41a not found: ID does not exist" containerID="ba6ea1660b010cf3b645c91f939fd765fce35a97ef7e60edad005bccd37cd41a" Nov 03 21:00:22.925944 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.925849 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba6ea1660b010cf3b645c91f939fd765fce35a97ef7e60edad005bccd37cd41a"} err="failed to get container status \"ba6ea1660b010cf3b645c91f939fd765fce35a97ef7e60edad005bccd37cd41a\": rpc error: code = NotFound desc = could not find container \"ba6ea1660b010cf3b645c91f939fd765fce35a97ef7e60edad005bccd37cd41a\": container with ID starting with ba6ea1660b010cf3b645c91f939fd765fce35a97ef7e60edad005bccd37cd41a not found: ID does not exist" Nov 03 21:00:22.925944 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.925867 5615 scope.go:122] "RemoveContainer" containerID="7e2ac422ad03848a80c894646ee9f674443af6032e2038a4d124828548d316ac" Nov 03 21:00:22.934466 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.934444 5615 scope.go:122] "RemoveContainer" containerID="7e2ac422ad03848a80c894646ee9f674443af6032e2038a4d124828548d316ac" Nov 03 21:00:22.934802 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:00:22.934780 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e2ac422ad03848a80c894646ee9f674443af6032e2038a4d124828548d316ac\": container with ID starting with 7e2ac422ad03848a80c894646ee9f674443af6032e2038a4d124828548d316ac not found: ID does not exist" containerID="7e2ac422ad03848a80c894646ee9f674443af6032e2038a4d124828548d316ac" Nov 03 21:00:22.934876 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:22.934811 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e2ac422ad03848a80c894646ee9f674443af6032e2038a4d124828548d316ac"} err="failed to get container status \"7e2ac422ad03848a80c894646ee9f674443af6032e2038a4d124828548d316ac\": rpc error: code = NotFound desc = could not find container \"7e2ac422ad03848a80c894646ee9f674443af6032e2038a4d124828548d316ac\": container with ID starting with 7e2ac422ad03848a80c894646ee9f674443af6032e2038a4d124828548d316ac not found: ID does not exist" Nov 03 21:00:23.069497 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.069464 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-10bfccd7-6998b5c8c9-x852r"] Nov 03 21:00:23.076228 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.076196 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-10bfccd7-6998b5c8c9-x852r"] Nov 03 21:00:23.179835 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.179803 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-10bfccd7-97649b7f8-66c64"] Nov 03 21:00:23.186393 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.186326 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-10bfccd7-97649b7f8-66c64"] Nov 03 21:00:23.339258 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.339225 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-320f60-57b9576c9-7frrd"] Nov 03 21:00:23.339662 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.339650 5615 cpu_manager.go:414] "containerMap: removing container" podUID="17c22d0b-c50f-4f98-a101-846557ad04f6" containerName="istio-proxy" Nov 03 21:00:23.339708 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.339663 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17c22d0b-c50f-4f98-a101-846557ad04f6" containerName="istio-proxy" Nov 03 21:00:23.339708 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.339679 5615 cpu_manager.go:414] "containerMap: removing container" podUID="88e07263-3ac7-4918-9b6f-234d0a5af683" containerName="payload-processing" Nov 03 21:00:23.339708 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.339684 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e07263-3ac7-4918-9b6f-234d0a5af683" containerName="payload-processing" Nov 03 21:00:23.339708 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.339698 5615 cpu_manager.go:414] "containerMap: removing container" podUID="17c22d0b-c50f-4f98-a101-846557ad04f6" containerName="istio-proxy" Nov 03 21:00:23.339708 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.339704 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17c22d0b-c50f-4f98-a101-846557ad04f6" containerName="istio-proxy" Nov 03 21:00:23.339852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.339714 5615 cpu_manager.go:414] "containerMap: removing container" podUID="1c889837-acc2-436c-8ede-741e552b47bc" containerName="payload-pre-processing" Nov 03 21:00:23.339852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.339718 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c889837-acc2-436c-8ede-741e552b47bc" containerName="payload-pre-processing" Nov 03 21:00:23.339852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.339734 5615 cpu_manager.go:414] "containerMap: removing container" podUID="95462c02-0404-47b2-bd2f-1fd27c989c20" containerName="payload-pre-processing" Nov 03 21:00:23.339852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.339739 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95462c02-0404-47b2-bd2f-1fd27c989c20" containerName="payload-pre-processing" Nov 03 21:00:23.339852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.339809 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="17c22d0b-c50f-4f98-a101-846557ad04f6" containerName="istio-proxy" Nov 03 21:00:23.339852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.339817 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c889837-acc2-436c-8ede-741e552b47bc" containerName="payload-pre-processing" Nov 03 21:00:23.339852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.339825 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="95462c02-0404-47b2-bd2f-1fd27c989c20" containerName="payload-pre-processing" Nov 03 21:00:23.339852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.339834 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e07263-3ac7-4918-9b6f-234d0a5af683" containerName="payload-processing" Nov 03 21:00:23.343441 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.343421 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-320f60-57b9576c9-7frrd" Nov 03 21:00:23.395101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.395050 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-320f60\"" Nov 03 21:00:23.395330 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.395305 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-320f60-dockercfg-kbvnx\"" Nov 03 21:00:23.396968 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.396941 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-320f60-57b9576c9-7frrd"] Nov 03 21:00:23.502152 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.502065 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hchm\" (UniqueName: \"kubernetes.io/projected/f3f0e38e-e1e7-451d-acc2-26df86d3c8ac-kube-api-access-8hchm\") pod \"payload-pre-processing-e2e-worker-w3-320f60-57b9576c9-7frrd\" (UID: \"f3f0e38e-e1e7-451d-acc2-26df86d3c8ac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-320f60-57b9576c9-7frrd" Nov 03 21:00:23.502459 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.502428 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3f0e38e-e1e7-451d-acc2-26df86d3c8ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-320f60-57b9576c9-7frrd\" (UID: \"f3f0e38e-e1e7-451d-acc2-26df86d3c8ac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-320f60-57b9576c9-7frrd" Nov 03 21:00:23.603618 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.603583 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hchm\" (UniqueName: \"kubernetes.io/projected/f3f0e38e-e1e7-451d-acc2-26df86d3c8ac-kube-api-access-8hchm\") pod \"payload-pre-processing-e2e-worker-w3-320f60-57b9576c9-7frrd\" (UID: \"f3f0e38e-e1e7-451d-acc2-26df86d3c8ac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-320f60-57b9576c9-7frrd" Nov 03 21:00:23.603766 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.603664 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3f0e38e-e1e7-451d-acc2-26df86d3c8ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-320f60-57b9576c9-7frrd\" (UID: \"f3f0e38e-e1e7-451d-acc2-26df86d3c8ac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-320f60-57b9576c9-7frrd" Nov 03 21:00:23.604424 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.604405 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3f0e38e-e1e7-451d-acc2-26df86d3c8ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-320f60-57b9576c9-7frrd\" (UID: \"f3f0e38e-e1e7-451d-acc2-26df86d3c8ac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-320f60-57b9576c9-7frrd" Nov 03 21:00:23.650489 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.650452 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3075b40b-4848-4e09-9d7e-a1b2eac692b1" path="/var/lib/kubelet/pods/3075b40b-4848-4e09-9d7e-a1b2eac692b1/volumes" Nov 03 21:00:23.650836 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.650821 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88e07263-3ac7-4918-9b6f-234d0a5af683" path="/var/lib/kubelet/pods/88e07263-3ac7-4918-9b6f-234d0a5af683/volumes" Nov 03 21:00:23.651169 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.651157 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95462c02-0404-47b2-bd2f-1fd27c989c20" path="/var/lib/kubelet/pods/95462c02-0404-47b2-bd2f-1fd27c989c20/volumes" Nov 03 21:00:23.675689 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.675661 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hchm\" (UniqueName: \"kubernetes.io/projected/f3f0e38e-e1e7-451d-acc2-26df86d3c8ac-kube-api-access-8hchm\") pod \"payload-pre-processing-e2e-worker-w3-320f60-57b9576c9-7frrd\" (UID: \"f3f0e38e-e1e7-451d-acc2-26df86d3c8ac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-320f60-57b9576c9-7frrd" Nov 03 21:00:23.927386 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.927289 5615 generic.go:356] "Generic (PLEG): container finished" podID="2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8" containerID="158baf6dbce07198386c9a7884e1ef8d83f58ef581ec9b17b24dc105020542d8" exitCode=60 Nov 03 21:00:23.927386 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.927326 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-tpxf2" event={"ID":"2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8","Type":"ContainerDied","Data":"158baf6dbce07198386c9a7884e1ef8d83f58ef581ec9b17b24dc105020542d8"} Nov 03 21:00:23.927386 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.927371 5615 scope.go:122] "RemoveContainer" containerID="871ca39471cec057d72c788a70c295aa0c44a7197287af84183a1f928fc49cab" Nov 03 21:00:23.927957 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.927737 5615 scope.go:122] "RemoveContainer" containerID="158baf6dbce07198386c9a7884e1ef8d83f58ef581ec9b17b24dc105020542d8" Nov 03 21:00:23.928049 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:00:23.928022 5615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895660-tpxf2_odh-ai-gateway-infra(2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-tpxf2" podUID="2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8" Nov 03 21:00:23.955556 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:23.955529 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-320f60-57b9576c9-7frrd" Nov 03 21:00:24.314265 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:24.314229 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc"] Nov 03 21:00:24.314461 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:24.314431 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" podUID="919ca8bf-b699-4bb0-88e1-7b71b2d5b11a" containerName="istio-proxy" containerID="cri-o://92c48ef302fee4f2027d54218d65e45f4656f8fc13e76846a353432ad5c722e4" gracePeriod=30 Nov 03 21:00:24.321406 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:24.321379 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-320f60-57b9576c9-7frrd"] Nov 03 21:00:24.362424 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:00:24.362393 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f0e38e_e1e7_451d_acc2_26df86d3c8ac.slice/crio-b6ce032235a2c3ba2c3b586f8940a5b45386cb4b231f50ca067d930c5539d68d WatchSource:0}: Error finding container b6ce032235a2c3ba2c3b586f8940a5b45386cb4b231f50ca067d930c5539d68d: Status 404 returned error can't find the container with id b6ce032235a2c3ba2c3b586f8940a5b45386cb4b231f50ca067d930c5539d68d Nov 03 21:00:24.935021 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:24.934987 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-320f60-57b9576c9-7frrd" event={"ID":"f3f0e38e-e1e7-451d-acc2-26df86d3c8ac","Type":"ContainerStarted","Data":"7196992cdbae6edaa1b0a98b4cfe85686852175aa616f7d44449e2a7e9a35445"} Nov 03 21:00:24.935433 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:24.935033 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-320f60-57b9576c9-7frrd" event={"ID":"f3f0e38e-e1e7-451d-acc2-26df86d3c8ac","Type":"ContainerStarted","Data":"b6ce032235a2c3ba2c3b586f8940a5b45386cb4b231f50ca067d930c5539d68d"} Nov 03 21:00:24.935433 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:24.935064 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-320f60-57b9576c9-7frrd" Nov 03 21:00:25.046124 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.046044 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-320f60-57b9576c9-7frrd" podStartSLOduration=2.046029344 podStartE2EDuration="2.046029344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:25.04450747 +0000 UTC m=+719.512286195" watchObservedRunningTime="2026-11-03 21:00:25.046029344 +0000 UTC m=+719.513808067" Nov 03 21:00:25.048573 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.048552 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-tpxf2"] Nov 03 21:00:25.194097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.194016 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-tpxf2" Nov 03 21:00:25.327743 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.327696 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8-maas-api-service-ca\") pod \"2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8\" (UID: \"2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8\") " Nov 03 21:00:25.327933 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.327747 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8-kube-api-access-ck7r7\" (UniqueName: \"kubernetes.io/projected/2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8-kube-api-access-ck7r7\") pod \"2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8\" (UID: \"2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8\") " Nov 03 21:00:25.328208 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.328183 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8-maas-api-service-ca" pod "2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8" (UID: "2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:25.331216 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.331179 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8-kube-api-access-ck7r7" pod "2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8" (UID: "2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8"). InnerVolumeSpecName "kube-api-access-ck7r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:25.428738 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.428706 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8-maas-api-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:25.428738 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.428735 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck7r7\" (UniqueName: \"kubernetes.io/projected/2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8-kube-api-access-ck7r7\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:25.672068 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.672035 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p"] Nov 03 21:00:25.672516 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.672502 5615 cpu_manager.go:414] "containerMap: removing container" podUID="2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8" containerName="cleanup" Nov 03 21:00:25.672581 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.672517 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8" containerName="cleanup" Nov 03 21:00:25.672581 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.672527 5615 cpu_manager.go:414] "containerMap: removing container" podUID="2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8" containerName="cleanup" Nov 03 21:00:25.672581 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.672532 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8" containerName="cleanup" Nov 03 21:00:25.672677 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.672615 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="17c22d0b-c50f-4f98-a101-846557ad04f6" containerName="istio-proxy" Nov 03 21:00:25.672677 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.672622 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8" containerName="cleanup" Nov 03 21:00:25.672677 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.672633 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8" containerName="cleanup" Nov 03 21:00:25.672677 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.672638 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8" containerName="cleanup" Nov 03 21:00:25.675626 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.675605 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:25.734801 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.734770 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p"] Nov 03 21:00:25.750736 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.750712 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-e84357-openshift-default-dockercfg-t4w57\"" Nov 03 21:00:25.832471 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.832438 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:25.832471 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.832481 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:25.832702 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.832516 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:25.832745 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.832686 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw6bh\" (UniqueName: \"kubernetes.io/projected/2a10a87e-4975-45cd-9f46-d9246dd3ae40-kube-api-access-mw6bh\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:25.832858 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.832830 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:25.832904 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.832873 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:25.833002 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.832980 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:25.833045 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.833026 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:25.833100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.833064 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:25.934777 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.934689 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:25.934777 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.934726 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:25.934777 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.934747 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:25.935107 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.934779 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw6bh\" (UniqueName: \"kubernetes.io/projected/2a10a87e-4975-45cd-9f46-d9246dd3ae40-kube-api-access-mw6bh\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:25.935107 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.934850 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:25.935107 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.934872 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:25.935107 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.934913 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:25.935107 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.934945 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:25.935689 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.935194 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:25.935689 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.935203 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:25.935689 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.935283 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:25.935689 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.935559 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:25.935689 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.935601 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:25.935866 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.935817 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:25.937347 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.937320 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:25.937800 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.937779 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:25.941871 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.941847 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-tpxf2" Nov 03 21:00:25.941980 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.941874 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-tpxf2" event={"ID":"2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8","Type":"ContainerDied","Data":"7189ee64996ff2d3bef702dae3d11ec7095ea2747146f1183485e5324a4970b6"} Nov 03 21:00:25.941980 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.941966 5615 scope.go:122] "RemoveContainer" containerID="158baf6dbce07198386c9a7884e1ef8d83f58ef581ec9b17b24dc105020542d8" Nov 03 21:00:25.995522 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.995487 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:25.995681 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:25.995586 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw6bh\" (UniqueName: \"kubernetes.io/projected/2a10a87e-4975-45cd-9f46-d9246dd3ae40-kube-api-access-mw6bh\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:26.110015 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:26.109981 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-tpxf2"] Nov 03 21:00:26.117102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:26.117039 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-tpxf2"] Nov 03 21:00:26.288322 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:26.288286 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:26.715401 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:26.715324 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p"] Nov 03 21:00:26.719027 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:00:26.718998 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a10a87e_4975_45cd_9f46_d9246dd3ae40.slice/crio-22d72fb0f98ae0e640591ef84fc2ad613bb6d25be8e0989f2ab06d8750d45c82 WatchSource:0}: Error finding container 22d72fb0f98ae0e640591ef84fc2ad613bb6d25be8e0989f2ab06d8750d45c82: Status 404 returned error can't find the container with id 22d72fb0f98ae0e640591ef84fc2ad613bb6d25be8e0989f2ab06d8750d45c82 Nov 03 21:00:26.721349 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:26.721315 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:00:26.721447 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:26.721392 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:00:26.721447 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:26.721440 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:00:26.949835 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:26.949798 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" event={"ID":"2a10a87e-4975-45cd-9f46-d9246dd3ae40","Type":"ContainerStarted","Data":"f986576b349e1b26761aab07bd273735ee57328db214e15ca51196de238ca8d7"} Nov 03 21:00:26.950355 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:26.949854 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" event={"ID":"2a10a87e-4975-45cd-9f46-d9246dd3ae40","Type":"ContainerStarted","Data":"22d72fb0f98ae0e640591ef84fc2ad613bb6d25be8e0989f2ab06d8750d45c82"} Nov 03 21:00:27.067459 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:27.066168 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" podStartSLOduration=2.066147565 podStartE2EDuration="2.066147565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:27.064659791 +0000 UTC m=+721.532438515" watchObservedRunningTime="2026-11-03 21:00:27.066147565 +0000 UTC m=+721.533926292" Nov 03 21:00:27.289485 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:27.289452 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:27.294733 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:27.294706 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:27.650770 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:27.650736 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8" path="/var/lib/kubelet/pods/2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8/volumes" Nov 03 21:00:27.704366 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:27.704329 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc"] Nov 03 21:00:27.704556 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:27.704517 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc" podUID="a6093506-8a41-4f1e-ae12-09e9346f6dc5" containerName="maas-api" containerID="cri-o://e55342c07a3eec82568223a3a7ab75d188fa02c9ee3eb9bff416fc89d68d7286" gracePeriod=30 Nov 03 21:00:27.959324 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:27.959292 5615 generic.go:356] "Generic (PLEG): container finished" podID="a6093506-8a41-4f1e-ae12-09e9346f6dc5" containerID="e55342c07a3eec82568223a3a7ab75d188fa02c9ee3eb9bff416fc89d68d7286" exitCode=0 Nov 03 21:00:27.959678 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:27.959363 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc" event={"ID":"a6093506-8a41-4f1e-ae12-09e9346f6dc5","Type":"ContainerDied","Data":"e55342c07a3eec82568223a3a7ab75d188fa02c9ee3eb9bff416fc89d68d7286"} Nov 03 21:00:27.959678 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:27.959645 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:27.960766 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:27.960741 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:00:27.975962 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:27.975939 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc" Nov 03 21:00:27.981499 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:27.981476 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6093506-8a41-4f1e-ae12-09e9346f6dc5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6093506-8a41-4f1e-ae12-09e9346f6dc5-maas-api-tls\") pod \"a6093506-8a41-4f1e-ae12-09e9346f6dc5\" (UID: \"a6093506-8a41-4f1e-ae12-09e9346f6dc5\") " Nov 03 21:00:27.981629 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:27.981537 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6093506-8a41-4f1e-ae12-09e9346f6dc5-kube-api-access-g9k7k\" (UniqueName: \"kubernetes.io/projected/a6093506-8a41-4f1e-ae12-09e9346f6dc5-kube-api-access-g9k7k\") pod \"a6093506-8a41-4f1e-ae12-09e9346f6dc5\" (UID: \"a6093506-8a41-4f1e-ae12-09e9346f6dc5\") " Nov 03 21:00:27.981629 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:27.981577 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6093506-8a41-4f1e-ae12-09e9346f6dc5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6093506-8a41-4f1e-ae12-09e9346f6dc5-metrics-certs\") pod \"a6093506-8a41-4f1e-ae12-09e9346f6dc5\" (UID: \"a6093506-8a41-4f1e-ae12-09e9346f6dc5\") " Nov 03 21:00:27.983678 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:27.983645 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6093506-8a41-4f1e-ae12-09e9346f6dc5-maas-api-tls" pod "a6093506-8a41-4f1e-ae12-09e9346f6dc5" (UID: "a6093506-8a41-4f1e-ae12-09e9346f6dc5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:00:27.983781 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:27.983724 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6093506-8a41-4f1e-ae12-09e9346f6dc5-metrics-certs" pod "a6093506-8a41-4f1e-ae12-09e9346f6dc5" (UID: "a6093506-8a41-4f1e-ae12-09e9346f6dc5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:00:27.983781 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:27.983770 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6093506-8a41-4f1e-ae12-09e9346f6dc5-kube-api-access-g9k7k" pod "a6093506-8a41-4f1e-ae12-09e9346f6dc5" (UID: "a6093506-8a41-4f1e-ae12-09e9346f6dc5"). InnerVolumeSpecName "kube-api-access-g9k7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:28.103116 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:28.099553 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6093506-8a41-4f1e-ae12-09e9346f6dc5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6093506-8a41-4f1e-ae12-09e9346f6dc5-openshift-service-ca\") pod \"a6093506-8a41-4f1e-ae12-09e9346f6dc5\" (UID: \"a6093506-8a41-4f1e-ae12-09e9346f6dc5\") " Nov 03 21:00:28.103116 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:28.099933 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6093506-8a41-4f1e-ae12-09e9346f6dc5-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:28.103116 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:28.099958 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9k7k\" (UniqueName: \"kubernetes.io/projected/a6093506-8a41-4f1e-ae12-09e9346f6dc5-kube-api-access-g9k7k\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:28.103116 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:28.099975 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6093506-8a41-4f1e-ae12-09e9346f6dc5-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:28.103116 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:28.101361 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6093506-8a41-4f1e-ae12-09e9346f6dc5-openshift-service-ca" pod "a6093506-8a41-4f1e-ae12-09e9346f6dc5" (UID: "a6093506-8a41-4f1e-ae12-09e9346f6dc5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:28.200687 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:28.200648 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6093506-8a41-4f1e-ae12-09e9346f6dc5-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:28.965379 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:28.965345 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc" Nov 03 21:00:28.965379 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:28.965374 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc" event={"ID":"a6093506-8a41-4f1e-ae12-09e9346f6dc5","Type":"ContainerDied","Data":"ab23910daa25fb3a7672cf9183417bb23f34bc88a98178b128a61b695beed683"} Nov 03 21:00:28.965927 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:28.965442 5615 scope.go:122] "RemoveContainer" containerID="e55342c07a3eec82568223a3a7ab75d188fa02c9ee3eb9bff416fc89d68d7286" Nov 03 21:00:29.120742 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.120712 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc"] Nov 03 21:00:29.168103 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.168056 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5b32d5a5-6d59cc5896-7drrc"] Nov 03 21:00:29.287344 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.287308 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2"] Nov 03 21:00:29.287750 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.287737 5615 cpu_manager.go:414] "containerMap: removing container" podUID="2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8" containerName="cleanup" Nov 03 21:00:29.287795 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.287751 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fbb7cc2-ef9c-4f5b-93cc-0faf069109d8" containerName="cleanup" Nov 03 21:00:29.287795 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.287763 5615 cpu_manager.go:414] "containerMap: removing container" podUID="a6093506-8a41-4f1e-ae12-09e9346f6dc5" containerName="maas-api" Nov 03 21:00:29.287795 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.287768 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6093506-8a41-4f1e-ae12-09e9346f6dc5" containerName="maas-api" Nov 03 21:00:29.287891 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.287849 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6093506-8a41-4f1e-ae12-09e9346f6dc5" containerName="maas-api" Nov 03 21:00:29.291319 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.291297 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2" Nov 03 21:00:29.312338 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.312311 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/933ca863-b42a-4f84-9d67-b36634e0d9b4-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2\" (UID: \"933ca863-b42a-4f84-9d67-b36634e0d9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2" Nov 03 21:00:29.312503 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.312351 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/933ca863-b42a-4f84-9d67-b36634e0d9b4-metrics-certs\") pod \"maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2\" (UID: \"933ca863-b42a-4f84-9d67-b36634e0d9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2" Nov 03 21:00:29.312503 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.312432 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4dvt\" (UniqueName: \"kubernetes.io/projected/933ca863-b42a-4f84-9d67-b36634e0d9b4-kube-api-access-j4dvt\") pod \"maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2\" (UID: \"933ca863-b42a-4f84-9d67-b36634e0d9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2" Nov 03 21:00:29.312590 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.312507 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/933ca863-b42a-4f84-9d67-b36634e0d9b4-maas-api-tls\") pod \"maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2\" (UID: \"933ca863-b42a-4f84-9d67-b36634e0d9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2" Nov 03 21:00:29.340898 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.340871 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-320f60-serving-cert\"" Nov 03 21:00:29.347029 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.347005 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2"] Nov 03 21:00:29.413038 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.413002 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/933ca863-b42a-4f84-9d67-b36634e0d9b4-maas-api-tls\") pod \"maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2\" (UID: \"933ca863-b42a-4f84-9d67-b36634e0d9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2" Nov 03 21:00:29.413255 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:00:29.413118 5615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-serving-cert: secret "maas-api-e2e-worker-w3-320f60-serving-cert" not found Nov 03 21:00:29.413255 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.413183 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/933ca863-b42a-4f84-9d67-b36634e0d9b4-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2\" (UID: \"933ca863-b42a-4f84-9d67-b36634e0d9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2" Nov 03 21:00:29.413255 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:00:29.413198 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/933ca863-b42a-4f84-9d67-b36634e0d9b4-maas-api-tls podName:933ca863-b42a-4f84-9d67-b36634e0d9b4 nodeName:}" failed. No retries permitted until 2026-11-03 21:00:29.913181503 +0000 UTC m=+724.380960204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/933ca863-b42a-4f84-9d67-b36634e0d9b4-maas-api-tls") pod "maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2" (UID: "933ca863-b42a-4f84-9d67-b36634e0d9b4") : secret "maas-api-e2e-worker-w3-320f60-serving-cert" not found Nov 03 21:00:29.413255 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.413252 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/933ca863-b42a-4f84-9d67-b36634e0d9b4-metrics-certs\") pod \"maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2\" (UID: \"933ca863-b42a-4f84-9d67-b36634e0d9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2" Nov 03 21:00:29.413477 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.413300 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4dvt\" (UniqueName: \"kubernetes.io/projected/933ca863-b42a-4f84-9d67-b36634e0d9b4-kube-api-access-j4dvt\") pod \"maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2\" (UID: \"933ca863-b42a-4f84-9d67-b36634e0d9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2" Nov 03 21:00:29.413838 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.413814 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/933ca863-b42a-4f84-9d67-b36634e0d9b4-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2\" (UID: \"933ca863-b42a-4f84-9d67-b36634e0d9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2" Nov 03 21:00:29.415889 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.415864 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/933ca863-b42a-4f84-9d67-b36634e0d9b4-metrics-certs\") pod \"maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2\" (UID: \"933ca863-b42a-4f84-9d67-b36634e0d9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2" Nov 03 21:00:29.477266 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.477234 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4dvt\" (UniqueName: \"kubernetes.io/projected/933ca863-b42a-4f84-9d67-b36634e0d9b4-kube-api-access-j4dvt\") pod \"maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2\" (UID: \"933ca863-b42a-4f84-9d67-b36634e0d9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2" Nov 03 21:00:29.590056 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.590018 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 21:00:29.614887 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.614838 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istio-data\") pod \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " Nov 03 21:00:29.614887 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.614888 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-workload-certs\") pod \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " Nov 03 21:00:29.615158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.614929 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istiod-ca-cert\") pod \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " Nov 03 21:00:29.615158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.614979 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-workload-socket\") pod \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " Nov 03 21:00:29.615158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.615016 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istio-token\" (UniqueName: \"kubernetes.io/projected/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istio-token\") pod \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " Nov 03 21:00:29.615158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.615114 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-kube-api-access-lgkpc\" (UniqueName: \"kubernetes.io/projected/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-kube-api-access-lgkpc\") pod \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " Nov 03 21:00:29.615396 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.615176 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-credential-socket\") pod \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " Nov 03 21:00:29.615396 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.615230 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istio-podinfo\") pod \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " Nov 03 21:00:29.615396 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.615217 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istio-data" pod "919ca8bf-b699-4bb0-88e1-7b71b2d5b11a" (UID: "919ca8bf-b699-4bb0-88e1-7b71b2d5b11a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:29.615396 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.615279 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istio-envoy\") pod \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\" (UID: \"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a\") " Nov 03 21:00:29.615396 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.615312 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-workload-socket" pod "919ca8bf-b699-4bb0-88e1-7b71b2d5b11a" (UID: "919ca8bf-b699-4bb0-88e1-7b71b2d5b11a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:29.615396 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.615319 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-workload-certs" pod "919ca8bf-b699-4bb0-88e1-7b71b2d5b11a" (UID: "919ca8bf-b699-4bb0-88e1-7b71b2d5b11a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:29.615668 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.615437 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-credential-socket" pod "919ca8bf-b699-4bb0-88e1-7b71b2d5b11a" (UID: "919ca8bf-b699-4bb0-88e1-7b71b2d5b11a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:29.615721 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.615691 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:29.615721 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.615696 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istiod-ca-cert" pod "919ca8bf-b699-4bb0-88e1-7b71b2d5b11a" (UID: "919ca8bf-b699-4bb0-88e1-7b71b2d5b11a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:29.615721 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.615711 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:29.615877 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.615725 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:29.615877 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.615738 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:29.617384 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.617360 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istio-podinfo" pod "919ca8bf-b699-4bb0-88e1-7b71b2d5b11a" (UID: "919ca8bf-b699-4bb0-88e1-7b71b2d5b11a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:00:29.617384 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.617382 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istio-envoy" pod "919ca8bf-b699-4bb0-88e1-7b71b2d5b11a" (UID: "919ca8bf-b699-4bb0-88e1-7b71b2d5b11a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:29.617681 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.617661 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-kube-api-access-lgkpc" pod "919ca8bf-b699-4bb0-88e1-7b71b2d5b11a" (UID: "919ca8bf-b699-4bb0-88e1-7b71b2d5b11a"). InnerVolumeSpecName "kube-api-access-lgkpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:29.617681 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.617671 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istio-token" pod "919ca8bf-b699-4bb0-88e1-7b71b2d5b11a" (UID: "919ca8bf-b699-4bb0-88e1-7b71b2d5b11a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:29.650224 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.650193 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6093506-8a41-4f1e-ae12-09e9346f6dc5" path="/var/lib/kubelet/pods/a6093506-8a41-4f1e-ae12-09e9346f6dc5/volumes" Nov 03 21:00:29.718475 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.718438 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:29.718475 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.718463 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgkpc\" (UniqueName: \"kubernetes.io/projected/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-kube-api-access-lgkpc\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:29.718475 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.718473 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:29.718475 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.718484 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:29.718814 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.718494 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:29.920447 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.920347 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/933ca863-b42a-4f84-9d67-b36634e0d9b4-maas-api-tls\") pod \"maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2\" (UID: \"933ca863-b42a-4f84-9d67-b36634e0d9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2" Nov 03 21:00:29.922930 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.922897 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/933ca863-b42a-4f84-9d67-b36634e0d9b4-maas-api-tls\") pod \"maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2\" (UID: \"933ca863-b42a-4f84-9d67-b36634e0d9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2" Nov 03 21:00:29.980236 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.980184 5615 generic.go:356] "Generic (PLEG): container finished" podID="919ca8bf-b699-4bb0-88e1-7b71b2d5b11a" containerID="92c48ef302fee4f2027d54218d65e45f4656f8fc13e76846a353432ad5c722e4" exitCode=0 Nov 03 21:00:29.980773 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.980285 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" Nov 03 21:00:29.980773 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.980283 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" event={"ID":"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a","Type":"ContainerDied","Data":"92c48ef302fee4f2027d54218d65e45f4656f8fc13e76846a353432ad5c722e4"} Nov 03 21:00:29.980773 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.980443 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc" event={"ID":"919ca8bf-b699-4bb0-88e1-7b71b2d5b11a","Type":"ContainerDied","Data":"0192ed7dea206c19139fb30205c1c10b82a036c13c82099e73b700d3cb478e08"} Nov 03 21:00:29.980773 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.980473 5615 scope.go:122] "RemoveContainer" containerID="92c48ef302fee4f2027d54218d65e45f4656f8fc13e76846a353432ad5c722e4" Nov 03 21:00:29.990748 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:29.990722 5615 scope.go:122] "RemoveContainer" containerID="cf7cfda68fd242eed2efb9d883bbd42823fe21df4020c6887e791d92c0534616" Nov 03 21:00:30.000905 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:30.000878 5615 scope.go:122] "RemoveContainer" containerID="92c48ef302fee4f2027d54218d65e45f4656f8fc13e76846a353432ad5c722e4" Nov 03 21:00:30.001286 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:00:30.001265 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92c48ef302fee4f2027d54218d65e45f4656f8fc13e76846a353432ad5c722e4\": container with ID starting with 92c48ef302fee4f2027d54218d65e45f4656f8fc13e76846a353432ad5c722e4 not found: ID does not exist" containerID="92c48ef302fee4f2027d54218d65e45f4656f8fc13e76846a353432ad5c722e4" Nov 03 21:00:30.001359 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:30.001294 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92c48ef302fee4f2027d54218d65e45f4656f8fc13e76846a353432ad5c722e4"} err="failed to get container status \"92c48ef302fee4f2027d54218d65e45f4656f8fc13e76846a353432ad5c722e4\": rpc error: code = NotFound desc = could not find container \"92c48ef302fee4f2027d54218d65e45f4656f8fc13e76846a353432ad5c722e4\": container with ID starting with 92c48ef302fee4f2027d54218d65e45f4656f8fc13e76846a353432ad5c722e4 not found: ID does not exist" Nov 03 21:00:30.001359 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:30.001313 5615 scope.go:122] "RemoveContainer" containerID="cf7cfda68fd242eed2efb9d883bbd42823fe21df4020c6887e791d92c0534616" Nov 03 21:00:30.001550 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:00:30.001528 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf7cfda68fd242eed2efb9d883bbd42823fe21df4020c6887e791d92c0534616\": container with ID starting with cf7cfda68fd242eed2efb9d883bbd42823fe21df4020c6887e791d92c0534616 not found: ID does not exist" containerID="cf7cfda68fd242eed2efb9d883bbd42823fe21df4020c6887e791d92c0534616" Nov 03 21:00:30.001591 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:30.001553 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf7cfda68fd242eed2efb9d883bbd42823fe21df4020c6887e791d92c0534616"} err="failed to get container status \"cf7cfda68fd242eed2efb9d883bbd42823fe21df4020c6887e791d92c0534616\": rpc error: code = NotFound desc = could not find container \"cf7cfda68fd242eed2efb9d883bbd42823fe21df4020c6887e791d92c0534616\": container with ID starting with cf7cfda68fd242eed2efb9d883bbd42823fe21df4020c6887e791d92c0534616 not found: ID does not exist" Nov 03 21:00:30.137444 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:30.137412 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc"] Nov 03 21:00:30.140944 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:30.140917 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-10bfccd7-openshift-default-767ffb8df7-mlhzc"] Nov 03 21:00:30.202976 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:30.202895 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2" Nov 03 21:00:30.595098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:30.588411 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2"] Nov 03 21:00:30.988209 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:30.988177 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2" event={"ID":"933ca863-b42a-4f84-9d67-b36634e0d9b4","Type":"ContainerStarted","Data":"b42bfed1ad9d12a7f029323e2a176cdcefb663c47103e9ce9b33c551925502cd"} Nov 03 21:00:31.650585 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.650561 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="919ca8bf-b699-4bb0-88e1-7b71b2d5b11a" path="/var/lib/kubelet/pods/919ca8bf-b699-4bb0-88e1-7b71b2d5b11a/volumes" Nov 03 21:00:31.825647 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.825596 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5"] Nov 03 21:00:31.826109 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.826091 5615 cpu_manager.go:414] "containerMap: removing container" podUID="919ca8bf-b699-4bb0-88e1-7b71b2d5b11a" containerName="istio-proxy" Nov 03 21:00:31.826163 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.826110 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="919ca8bf-b699-4bb0-88e1-7b71b2d5b11a" containerName="istio-proxy" Nov 03 21:00:31.826163 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.826147 5615 cpu_manager.go:414] "containerMap: removing container" podUID="919ca8bf-b699-4bb0-88e1-7b71b2d5b11a" containerName="istio-proxy" Nov 03 21:00:31.826163 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.826155 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="919ca8bf-b699-4bb0-88e1-7b71b2d5b11a" containerName="istio-proxy" Nov 03 21:00:31.826251 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.826235 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="919ca8bf-b699-4bb0-88e1-7b71b2d5b11a" containerName="istio-proxy" Nov 03 21:00:31.826251 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.826250 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="919ca8bf-b699-4bb0-88e1-7b71b2d5b11a" containerName="istio-proxy" Nov 03 21:00:31.828818 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.828794 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:31.841388 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.841359 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:31.841508 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.841394 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/884c8308-cf0b-496d-9f55-bb6563fe0238-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:31.841508 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.841427 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c8xv\" (UniqueName: \"kubernetes.io/projected/884c8308-cf0b-496d-9f55-bb6563fe0238-kube-api-access-2c8xv\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:31.841594 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.841518 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:31.841594 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.841580 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/884c8308-cf0b-496d-9f55-bb6563fe0238-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:31.841666 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.841649 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:31.841697 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.841680 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:31.841731 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.841717 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/884c8308-cf0b-496d-9f55-bb6563fe0238-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:31.841802 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.841790 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:31.878511 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.878420 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-791c32-openshift-default-dockercfg-zfm58\"" Nov 03 21:00:31.927763 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.927729 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5"] Nov 03 21:00:31.942540 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.942503 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/884c8308-cf0b-496d-9f55-bb6563fe0238-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:31.942742 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.942559 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:31.942742 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.942582 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:31.942742 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.942615 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/884c8308-cf0b-496d-9f55-bb6563fe0238-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:31.942742 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.942645 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:31.942742 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.942668 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:31.942742 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.942690 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/884c8308-cf0b-496d-9f55-bb6563fe0238-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:31.943032 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.942743 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c8xv\" (UniqueName: \"kubernetes.io/projected/884c8308-cf0b-496d-9f55-bb6563fe0238-kube-api-access-2c8xv\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:31.943032 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.942800 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:31.943182 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.943103 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:31.943182 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.943148 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:31.943259 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.943224 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:31.943395 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.943377 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:31.943618 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.943601 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/884c8308-cf0b-496d-9f55-bb6563fe0238-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:31.945106 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.945064 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:31.945742 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:31.945720 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/884c8308-cf0b-496d-9f55-bb6563fe0238-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:32.016099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:32.015898 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2" event={"ID":"933ca863-b42a-4f84-9d67-b36634e0d9b4","Type":"ContainerStarted","Data":"dff160c8f699bd8be9082207033d5790be32035bd1d808d3d698cbd0ac178842"} Nov 03 21:00:32.036099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:32.016858 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2" Nov 03 21:00:32.036099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:32.020626 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/884c8308-cf0b-496d-9f55-bb6563fe0238-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:32.036099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:32.022038 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c8xv\" (UniqueName: \"kubernetes.io/projected/884c8308-cf0b-496d-9f55-bb6563fe0238-kube-api-access-2c8xv\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:32.129190 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:32.129044 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2" podStartSLOduration=2.14409507 podStartE2EDuration="3.129026689s" totalImagesPullingTime="984.931619ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:32.128186863 +0000 UTC m=+726.595965583" watchObservedRunningTime="2026-11-03 21:00:32.129026689 +0000 UTC m=+726.596805413" Nov 03 21:00:32.141254 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:32.141217 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:32.556310 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:32.556284 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5"] Nov 03 21:00:32.558123 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:00:32.558059 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod884c8308_cf0b_496d_9f55_bb6563fe0238.slice/crio-b426577046db478a746af3b9734d2c67daea67783af0ef2209b701d22c169cd6 WatchSource:0}: Error finding container b426577046db478a746af3b9734d2c67daea67783af0ef2209b701d22c169cd6: Status 404 returned error can't find the container with id b426577046db478a746af3b9734d2c67daea67783af0ef2209b701d22c169cd6 Nov 03 21:00:32.560556 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:32.560517 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:00:32.560646 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:32.560586 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:00:32.560646 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:32.560617 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:00:33.023362 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:33.023324 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" event={"ID":"884c8308-cf0b-496d-9f55-bb6563fe0238","Type":"ContainerStarted","Data":"46727b5c84e14613cb6fd6dbdc300439f0b7bba72f858cbd7927e91592837e79"} Nov 03 21:00:33.023791 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:33.023386 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" event={"ID":"884c8308-cf0b-496d-9f55-bb6563fe0238","Type":"ContainerStarted","Data":"b426577046db478a746af3b9734d2c67daea67783af0ef2209b701d22c169cd6"} Nov 03 21:00:33.139204 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:33.139150 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" podStartSLOduration=2.139135919 podStartE2EDuration="2.139135919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:33.137630295 +0000 UTC m=+727.605409023" watchObservedRunningTime="2026-11-03 21:00:33.139135919 +0000 UTC m=+727.606914642" Nov 03 21:00:33.142025 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:33.142002 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:33.147443 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:33.147419 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:34.028520 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:34.028491 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:34.029511 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:34.029485 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:00:34.442205 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:34.442126 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt"] Nov 03 21:00:34.445254 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:34.445229 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt" Nov 03 21:00:34.467535 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:34.467500 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb26n\" (UniqueName: \"kubernetes.io/projected/b037fb5a-8768-4e13-b190-b5226818fd81-kube-api-access-nb26n\") pod \"maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt\" (UID: \"b037fb5a-8768-4e13-b190-b5226818fd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt" Nov 03 21:00:34.467676 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:34.467576 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b037fb5a-8768-4e13-b190-b5226818fd81-metrics-certs\") pod \"maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt\" (UID: \"b037fb5a-8768-4e13-b190-b5226818fd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt" Nov 03 21:00:34.467732 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:34.467697 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b037fb5a-8768-4e13-b190-b5226818fd81-maas-api-tls\") pod \"maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt\" (UID: \"b037fb5a-8768-4e13-b190-b5226818fd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt" Nov 03 21:00:34.467770 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:34.467727 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b037fb5a-8768-4e13-b190-b5226818fd81-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt\" (UID: \"b037fb5a-8768-4e13-b190-b5226818fd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt" Nov 03 21:00:34.494477 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:34.494443 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-099a5e-serving-cert\"" Nov 03 21:00:34.501237 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:34.501208 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt"] Nov 03 21:00:34.568320 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:34.568289 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb26n\" (UniqueName: \"kubernetes.io/projected/b037fb5a-8768-4e13-b190-b5226818fd81-kube-api-access-nb26n\") pod \"maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt\" (UID: \"b037fb5a-8768-4e13-b190-b5226818fd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt" Nov 03 21:00:34.568531 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:34.568352 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b037fb5a-8768-4e13-b190-b5226818fd81-metrics-certs\") pod \"maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt\" (UID: \"b037fb5a-8768-4e13-b190-b5226818fd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt" Nov 03 21:00:34.568531 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:34.568430 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b037fb5a-8768-4e13-b190-b5226818fd81-maas-api-tls\") pod \"maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt\" (UID: \"b037fb5a-8768-4e13-b190-b5226818fd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt" Nov 03 21:00:34.568531 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:34.568452 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b037fb5a-8768-4e13-b190-b5226818fd81-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt\" (UID: \"b037fb5a-8768-4e13-b190-b5226818fd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt" Nov 03 21:00:34.568531 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:00:34.568521 5615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-serving-cert: secret "maas-api-e2e-worker-w2-099a5e-serving-cert" not found Nov 03 21:00:34.568721 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:00:34.568596 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b037fb5a-8768-4e13-b190-b5226818fd81-maas-api-tls podName:b037fb5a-8768-4e13-b190-b5226818fd81 nodeName:}" failed. No retries permitted until 2026-11-03 21:00:35.068576012 +0000 UTC m=+729.536354735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b037fb5a-8768-4e13-b190-b5226818fd81-maas-api-tls") pod "maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt" (UID: "b037fb5a-8768-4e13-b190-b5226818fd81") : secret "maas-api-e2e-worker-w2-099a5e-serving-cert" not found Nov 03 21:00:34.569220 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:34.569197 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b037fb5a-8768-4e13-b190-b5226818fd81-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt\" (UID: \"b037fb5a-8768-4e13-b190-b5226818fd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt" Nov 03 21:00:34.571193 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:34.571168 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b037fb5a-8768-4e13-b190-b5226818fd81-metrics-certs\") pod \"maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt\" (UID: \"b037fb5a-8768-4e13-b190-b5226818fd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt" Nov 03 21:00:34.638776 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:34.638745 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb26n\" (UniqueName: \"kubernetes.io/projected/b037fb5a-8768-4e13-b190-b5226818fd81-kube-api-access-nb26n\") pod \"maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt\" (UID: \"b037fb5a-8768-4e13-b190-b5226818fd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt" Nov 03 21:00:35.076788 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:35.076755 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b037fb5a-8768-4e13-b190-b5226818fd81-maas-api-tls\") pod \"maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt\" (UID: \"b037fb5a-8768-4e13-b190-b5226818fd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt" Nov 03 21:00:35.079175 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:35.079154 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b037fb5a-8768-4e13-b190-b5226818fd81-maas-api-tls\") pod \"maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt\" (UID: \"b037fb5a-8768-4e13-b190-b5226818fd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt" Nov 03 21:00:35.357948 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:35.357834 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt" Nov 03 21:00:35.730137 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:35.730099 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt"] Nov 03 21:00:35.947677 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:35.947641 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-320f60-57b9576c9-7frrd" Nov 03 21:00:36.047320 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:36.047276 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt" event={"ID":"b037fb5a-8768-4e13-b190-b5226818fd81","Type":"ContainerStarted","Data":"7da20b6dbcdae452eb3333632e233fdf547587bf2ca8aa274006714f5298639f"} Nov 03 21:00:37.053733 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:37.053699 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt" event={"ID":"b037fb5a-8768-4e13-b190-b5226818fd81","Type":"ContainerStarted","Data":"86e4d83bf830865c8b7d34e058e47f23a3148758b4526ba9b9fbaf7850340a03"} Nov 03 21:00:37.054173 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:37.053766 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt" Nov 03 21:00:37.168969 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:37.168914 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt" podStartSLOduration=2.280211707 podStartE2EDuration="3.168898996s" totalImagesPullingTime="888.687289ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:37.167931982 +0000 UTC m=+731.635710707" watchObservedRunningTime="2026-11-03 21:00:37.168898996 +0000 UTC m=+731.636677718" Nov 03 21:00:39.034242 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:39.034212 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2" Nov 03 21:00:40.642881 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:00:40.642842 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:00:43.066266 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:43.066231 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt" Nov 03 21:00:46.103707 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:46.103672 5615 generic.go:356] "Generic (PLEG): container finished" podID="11b3bf8b-24eb-4b93-80a1-fabdf14cff21" containerID="5da03baf1f33f94c77a73c9a13a54392fcf186b52f9c17a702b785901e25abd4" exitCode=0 Nov 03 21:00:46.104158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:46.103726 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" event={"ID":"11b3bf8b-24eb-4b93-80a1-fabdf14cff21","Type":"ContainerDied","Data":"5da03baf1f33f94c77a73c9a13a54392fcf186b52f9c17a702b785901e25abd4"} Nov 03 21:00:46.104158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:46.104128 5615 scope.go:122] "RemoveContainer" containerID="5da03baf1f33f94c77a73c9a13a54392fcf186b52f9c17a702b785901e25abd4" Nov 03 21:00:46.105352 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:46.105324 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:00:46.105410 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:46.105390 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:00:46.105444 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:46.105420 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:00:46.876891 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:46.876851 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:47.069185 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:47.069152 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc"] Nov 03 21:00:47.069415 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:47.069386 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" podUID="1b8f7b11-df18-4a7b-876a-a1ad15c10882" containerName="istio-proxy" containerID="cri-o://28c0713135daac31f9a02fba48a531f81486c0be7590574ccff5659ab65b902e" gracePeriod=30 Nov 03 21:00:47.130934 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:47.130838 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" event={"ID":"11b3bf8b-24eb-4b93-80a1-fabdf14cff21","Type":"ContainerStarted","Data":"6a95a3fe50fd2cf9bd879ce6e5cbb94c7cdce5cac048311030c45c169c135277"} Nov 03 21:00:48.084870 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:48.084837 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:48.085992 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:48.085965 5615 patch_prober.go:29] interesting pod/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 21:00:48.086149 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:48.086014 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" podUID="11b3bf8b-24eb-4b93-80a1-fabdf14cff21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 21:00:49.084926 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:49.084876 5615 patch_prober.go:29] interesting pod/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 21:00:49.085355 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:49.084939 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" podUID="11b3bf8b-24eb-4b93-80a1-fabdf14cff21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 21:00:50.085029 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:50.084987 5615 patch_prober.go:29] interesting pod/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 21:00:50.085468 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:50.085049 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" podUID="11b3bf8b-24eb-4b93-80a1-fabdf14cff21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 21:00:51.084672 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:51.084635 5615 patch_prober.go:29] interesting pod/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 21:00:51.084871 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:51.084690 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" podUID="11b3bf8b-24eb-4b93-80a1-fabdf14cff21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 21:00:52.084895 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:52.084862 5615 patch_prober.go:29] interesting pod/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 21:00:52.085228 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:52.084927 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" podUID="11b3bf8b-24eb-4b93-80a1-fabdf14cff21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 21:00:52.381179 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:52.381147 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 21:00:52.522653 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:52.522620 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istio-token\" (UniqueName: \"kubernetes.io/projected/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istio-token\") pod \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " Nov 03 21:00:52.522852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:52.522678 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istio-envoy\") pod \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " Nov 03 21:00:52.522852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:52.522723 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b8f7b11-df18-4a7b-876a-a1ad15c10882-kube-api-access-qtxkm\" (UniqueName: \"kubernetes.io/projected/1b8f7b11-df18-4a7b-876a-a1ad15c10882-kube-api-access-qtxkm\") pod \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " Nov 03 21:00:52.522852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:52.522756 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-credential-socket\") pod \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " Nov 03 21:00:52.522852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:52.522789 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-workload-socket\") pod \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " Nov 03 21:00:52.523121 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:52.522887 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istiod-ca-cert\") pod \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " Nov 03 21:00:52.523121 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:52.522989 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-workload-certs\") pod \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " Nov 03 21:00:52.523121 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:52.523029 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istio-data\") pod \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " Nov 03 21:00:52.523121 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:52.523089 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istio-podinfo\") pod \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\" (UID: \"1b8f7b11-df18-4a7b-876a-a1ad15c10882\") " Nov 03 21:00:52.523958 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:52.523574 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istiod-ca-cert" pod "1b8f7b11-df18-4a7b-876a-a1ad15c10882" (UID: "1b8f7b11-df18-4a7b-876a-a1ad15c10882"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:52.523958 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:52.523884 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-workload-socket" pod "1b8f7b11-df18-4a7b-876a-a1ad15c10882" (UID: "1b8f7b11-df18-4a7b-876a-a1ad15c10882"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:52.523958 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:52.523902 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-workload-certs" pod "1b8f7b11-df18-4a7b-876a-a1ad15c10882" (UID: "1b8f7b11-df18-4a7b-876a-a1ad15c10882"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:52.523958 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:52.523928 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istio-data" pod "1b8f7b11-df18-4a7b-876a-a1ad15c10882" (UID: "1b8f7b11-df18-4a7b-876a-a1ad15c10882"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:52.524284 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:52.524004 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-credential-socket" pod "1b8f7b11-df18-4a7b-876a-a1ad15c10882" (UID: "1b8f7b11-df18-4a7b-876a-a1ad15c10882"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:52.525271 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:52.525240 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istio-token" pod "1b8f7b11-df18-4a7b-876a-a1ad15c10882" (UID: "1b8f7b11-df18-4a7b-876a-a1ad15c10882"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:52.525672 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:52.525645 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istio-podinfo" pod "1b8f7b11-df18-4a7b-876a-a1ad15c10882" (UID: "1b8f7b11-df18-4a7b-876a-a1ad15c10882"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:00:52.525813 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:52.525782 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istio-envoy" pod "1b8f7b11-df18-4a7b-876a-a1ad15c10882" (UID: "1b8f7b11-df18-4a7b-876a-a1ad15c10882"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:52.525974 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:52.525953 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b8f7b11-df18-4a7b-876a-a1ad15c10882-kube-api-access-qtxkm" pod "1b8f7b11-df18-4a7b-876a-a1ad15c10882" (UID: "1b8f7b11-df18-4a7b-876a-a1ad15c10882"). InnerVolumeSpecName "kube-api-access-qtxkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:52.624180 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:52.624094 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:52.624180 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:52.624126 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:52.624180 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:52.624136 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:52.624180 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:52.624146 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:52.624180 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:52.624154 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:52.624180 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:52.624164 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:52.624180 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:52.624173 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtxkm\" (UniqueName: \"kubernetes.io/projected/1b8f7b11-df18-4a7b-876a-a1ad15c10882-kube-api-access-qtxkm\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:52.624180 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:52.624183 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:52.624546 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:52.624192 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8f7b11-df18-4a7b-876a-a1ad15c10882-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:53.088909 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:53.088878 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:53.089340 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:53.089164 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:53.089941 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:53.089923 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:00:53.161897 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:53.161863 5615 generic.go:356] "Generic (PLEG): container finished" podID="1b8f7b11-df18-4a7b-876a-a1ad15c10882" containerID="28c0713135daac31f9a02fba48a531f81486c0be7590574ccff5659ab65b902e" exitCode=0 Nov 03 21:00:53.162091 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:53.161908 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" Nov 03 21:00:53.162091 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:53.161949 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" event={"ID":"1b8f7b11-df18-4a7b-876a-a1ad15c10882","Type":"ContainerDied","Data":"28c0713135daac31f9a02fba48a531f81486c0be7590574ccff5659ab65b902e"} Nov 03 21:00:53.162091 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:53.162003 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc" event={"ID":"1b8f7b11-df18-4a7b-876a-a1ad15c10882","Type":"ContainerDied","Data":"bb897bdf499ffc5de3519992b0c2bf2197c9a04265d4a5c0bdd69da4973a8160"} Nov 03 21:00:53.162091 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:53.162028 5615 scope.go:122] "RemoveContainer" containerID="28c0713135daac31f9a02fba48a531f81486c0be7590574ccff5659ab65b902e" Nov 03 21:00:53.171935 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:53.171914 5615 scope.go:122] "RemoveContainer" containerID="c1264547ba64c2072b9204fabb4331a36d7ae9910ffed931f62d8358d162832f" Nov 03 21:00:53.180153 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:53.180131 5615 scope.go:122] "RemoveContainer" containerID="28c0713135daac31f9a02fba48a531f81486c0be7590574ccff5659ab65b902e" Nov 03 21:00:53.180466 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:00:53.180442 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28c0713135daac31f9a02fba48a531f81486c0be7590574ccff5659ab65b902e\": container with ID starting with 28c0713135daac31f9a02fba48a531f81486c0be7590574ccff5659ab65b902e not found: ID does not exist" containerID="28c0713135daac31f9a02fba48a531f81486c0be7590574ccff5659ab65b902e" Nov 03 21:00:53.180561 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:53.180471 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28c0713135daac31f9a02fba48a531f81486c0be7590574ccff5659ab65b902e"} err="failed to get container status \"28c0713135daac31f9a02fba48a531f81486c0be7590574ccff5659ab65b902e\": rpc error: code = NotFound desc = could not find container \"28c0713135daac31f9a02fba48a531f81486c0be7590574ccff5659ab65b902e\": container with ID starting with 28c0713135daac31f9a02fba48a531f81486c0be7590574ccff5659ab65b902e not found: ID does not exist" Nov 03 21:00:53.180561 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:53.180490 5615 scope.go:122] "RemoveContainer" containerID="c1264547ba64c2072b9204fabb4331a36d7ae9910ffed931f62d8358d162832f" Nov 03 21:00:53.180793 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:00:53.180774 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1264547ba64c2072b9204fabb4331a36d7ae9910ffed931f62d8358d162832f\": container with ID starting with c1264547ba64c2072b9204fabb4331a36d7ae9910ffed931f62d8358d162832f not found: ID does not exist" containerID="c1264547ba64c2072b9204fabb4331a36d7ae9910ffed931f62d8358d162832f" Nov 03 21:00:53.180837 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:53.180801 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1264547ba64c2072b9204fabb4331a36d7ae9910ffed931f62d8358d162832f"} err="failed to get container status \"c1264547ba64c2072b9204fabb4331a36d7ae9910ffed931f62d8358d162832f\": rpc error: code = NotFound desc = could not find container \"c1264547ba64c2072b9204fabb4331a36d7ae9910ffed931f62d8358d162832f\": container with ID starting with c1264547ba64c2072b9204fabb4331a36d7ae9910ffed931f62d8358d162832f not found: ID does not exist" Nov 03 21:00:53.331234 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:53.331190 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc"] Nov 03 21:00:53.331903 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:53.331874 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-c9f7f443-openshift-default-74cbbfcbcc-k97nc"] Nov 03 21:00:53.651306 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:53.651272 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b8f7b11-df18-4a7b-876a-a1ad15c10882" path="/var/lib/kubelet/pods/1b8f7b11-df18-4a7b-876a-a1ad15c10882/volumes" Nov 03 21:00:59.001296 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:59.001251 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:00:59.001781 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:59.001319 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" podUID="2a10a87e-4975-45cd-9f46-d9246dd3ae40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:59.909118 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:59.909039 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr"] Nov 03 21:00:59.909611 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:59.909587 5615 cpu_manager.go:414] "containerMap: removing container" podUID="1b8f7b11-df18-4a7b-876a-a1ad15c10882" containerName="istio-proxy" Nov 03 21:00:59.909611 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:59.909608 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b8f7b11-df18-4a7b-876a-a1ad15c10882" containerName="istio-proxy" Nov 03 21:00:59.909790 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:59.909621 5615 cpu_manager.go:414] "containerMap: removing container" podUID="1b8f7b11-df18-4a7b-876a-a1ad15c10882" containerName="istio-proxy" Nov 03 21:00:59.909790 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:59.909626 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b8f7b11-df18-4a7b-876a-a1ad15c10882" containerName="istio-proxy" Nov 03 21:00:59.909790 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:59.909693 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b8f7b11-df18-4a7b-876a-a1ad15c10882" containerName="istio-proxy" Nov 03 21:00:59.909790 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:59.909702 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b8f7b11-df18-4a7b-876a-a1ad15c10882" containerName="istio-proxy" Nov 03 21:00:59.912863 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:00:59.912845 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr" Nov 03 21:01:00.010099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.009435 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfhxk\" (UniqueName: \"kubernetes.io/projected/ce51a38f-47a9-4926-adec-cecfceba6000-kube-api-access-gfhxk\") pod \"maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr\" (UID: \"ce51a38f-47a9-4926-adec-cecfceba6000\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr" Nov 03 21:01:00.010099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.009570 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-791c32-serving-cert\"" Nov 03 21:01:00.010099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.009594 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce51a38f-47a9-4926-adec-cecfceba6000-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr\" (UID: \"ce51a38f-47a9-4926-adec-cecfceba6000\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr" Nov 03 21:01:00.010099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.009629 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce51a38f-47a9-4926-adec-cecfceba6000-maas-api-tls\") pod \"maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr\" (UID: \"ce51a38f-47a9-4926-adec-cecfceba6000\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr" Nov 03 21:01:00.010099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.009654 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce51a38f-47a9-4926-adec-cecfceba6000-metrics-certs\") pod \"maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr\" (UID: \"ce51a38f-47a9-4926-adec-cecfceba6000\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr" Nov 03 21:01:00.012314 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.012285 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr"] Nov 03 21:01:00.110228 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.110192 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce51a38f-47a9-4926-adec-cecfceba6000-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr\" (UID: \"ce51a38f-47a9-4926-adec-cecfceba6000\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr" Nov 03 21:01:00.110424 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.110234 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce51a38f-47a9-4926-adec-cecfceba6000-maas-api-tls\") pod \"maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr\" (UID: \"ce51a38f-47a9-4926-adec-cecfceba6000\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr" Nov 03 21:01:00.110424 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.110261 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce51a38f-47a9-4926-adec-cecfceba6000-metrics-certs\") pod \"maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr\" (UID: \"ce51a38f-47a9-4926-adec-cecfceba6000\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr" Nov 03 21:01:00.110424 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.110325 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfhxk\" (UniqueName: \"kubernetes.io/projected/ce51a38f-47a9-4926-adec-cecfceba6000-kube-api-access-gfhxk\") pod \"maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr\" (UID: \"ce51a38f-47a9-4926-adec-cecfceba6000\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr" Nov 03 21:01:00.110424 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:01:00.110412 5615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-serving-cert: secret "maas-api-e2e-worker-w0-791c32-serving-cert" not found Nov 03 21:01:00.110626 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:01:00.110495 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce51a38f-47a9-4926-adec-cecfceba6000-maas-api-tls podName:ce51a38f-47a9-4926-adec-cecfceba6000 nodeName:}" failed. No retries permitted until 2026-11-03 21:01:00.610475339 +0000 UTC m=+755.078254064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ce51a38f-47a9-4926-adec-cecfceba6000-maas-api-tls") pod "maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr" (UID: "ce51a38f-47a9-4926-adec-cecfceba6000") : secret "maas-api-e2e-worker-w0-791c32-serving-cert" not found Nov 03 21:01:00.110866 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.110845 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce51a38f-47a9-4926-adec-cecfceba6000-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr\" (UID: \"ce51a38f-47a9-4926-adec-cecfceba6000\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr" Nov 03 21:01:00.113033 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.113009 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce51a38f-47a9-4926-adec-cecfceba6000-metrics-certs\") pod \"maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr\" (UID: \"ce51a38f-47a9-4926-adec-cecfceba6000\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr" Nov 03 21:01:00.169653 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.169578 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfhxk\" (UniqueName: \"kubernetes.io/projected/ce51a38f-47a9-4926-adec-cecfceba6000-kube-api-access-gfhxk\") pod \"maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr\" (UID: \"ce51a38f-47a9-4926-adec-cecfceba6000\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr" Nov 03 21:01:00.277897 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.277855 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a41ad0-5458b6d47-62h44"] Nov 03 21:01:00.281944 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.281912 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a41ad0-5458b6d47-62h44" Nov 03 21:01:00.311728 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.311686 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38461555-612e-478c-a9f8-98807b526d0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a41ad0-5458b6d47-62h44\" (UID: \"38461555-612e-478c-a9f8-98807b526d0f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a41ad0-5458b6d47-62h44" Nov 03 21:01:00.311873 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.311833 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnn85\" (UniqueName: \"kubernetes.io/projected/38461555-612e-478c-a9f8-98807b526d0f-kube-api-access-nnn85\") pod \"payload-pre-processing-e2e-worker-w1-a41ad0-5458b6d47-62h44\" (UID: \"38461555-612e-478c-a9f8-98807b526d0f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a41ad0-5458b6d47-62h44" Nov 03 21:01:00.333860 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.333800 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-a41ad0\"" Nov 03 21:01:00.333860 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.333840 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-a41ad0-dockercfg-drgs7\"" Nov 03 21:01:00.336512 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.336482 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a41ad0-5458b6d47-62h44"] Nov 03 21:01:00.412444 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.412406 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38461555-612e-478c-a9f8-98807b526d0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a41ad0-5458b6d47-62h44\" (UID: \"38461555-612e-478c-a9f8-98807b526d0f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a41ad0-5458b6d47-62h44" Nov 03 21:01:00.412613 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.412495 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnn85\" (UniqueName: \"kubernetes.io/projected/38461555-612e-478c-a9f8-98807b526d0f-kube-api-access-nnn85\") pod \"payload-pre-processing-e2e-worker-w1-a41ad0-5458b6d47-62h44\" (UID: \"38461555-612e-478c-a9f8-98807b526d0f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a41ad0-5458b6d47-62h44" Nov 03 21:01:00.413175 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.413150 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38461555-612e-478c-a9f8-98807b526d0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a41ad0-5458b6d47-62h44\" (UID: \"38461555-612e-478c-a9f8-98807b526d0f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a41ad0-5458b6d47-62h44" Nov 03 21:01:00.487054 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.483701 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a41ad0-8665dcdf7d-948h2"] Nov 03 21:01:00.491172 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.491012 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a41ad0-8665dcdf7d-948h2" Nov 03 21:01:00.505229 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.503306 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnn85\" (UniqueName: \"kubernetes.io/projected/38461555-612e-478c-a9f8-98807b526d0f-kube-api-access-nnn85\") pod \"payload-pre-processing-e2e-worker-w1-a41ad0-5458b6d47-62h44\" (UID: \"38461555-612e-478c-a9f8-98807b526d0f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a41ad0-5458b6d47-62h44" Nov 03 21:01:00.515054 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.515015 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v55ss\" (UniqueName: \"kubernetes.io/projected/5611d11d-6bc4-4725-a5e7-d2cf9203f5ce-kube-api-access-v55ss\") pod \"payload-processing-e2e-worker-w1-a41ad0-8665dcdf7d-948h2\" (UID: \"5611d11d-6bc4-4725-a5e7-d2cf9203f5ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a41ad0-8665dcdf7d-948h2" Nov 03 21:01:00.515235 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.515204 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5611d11d-6bc4-4725-a5e7-d2cf9203f5ce-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a41ad0-8665dcdf7d-948h2\" (UID: \"5611d11d-6bc4-4725-a5e7-d2cf9203f5ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a41ad0-8665dcdf7d-948h2" Nov 03 21:01:00.532208 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.532176 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a41ad0-8665dcdf7d-948h2"] Nov 03 21:01:00.595195 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.595149 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a41ad0-5458b6d47-62h44" Nov 03 21:01:00.616277 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.616240 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce51a38f-47a9-4926-adec-cecfceba6000-maas-api-tls\") pod \"maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr\" (UID: \"ce51a38f-47a9-4926-adec-cecfceba6000\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr" Nov 03 21:01:00.616424 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.616361 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5611d11d-6bc4-4725-a5e7-d2cf9203f5ce-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a41ad0-8665dcdf7d-948h2\" (UID: \"5611d11d-6bc4-4725-a5e7-d2cf9203f5ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a41ad0-8665dcdf7d-948h2" Nov 03 21:01:00.616520 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.616446 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v55ss\" (UniqueName: \"kubernetes.io/projected/5611d11d-6bc4-4725-a5e7-d2cf9203f5ce-kube-api-access-v55ss\") pod \"payload-processing-e2e-worker-w1-a41ad0-8665dcdf7d-948h2\" (UID: \"5611d11d-6bc4-4725-a5e7-d2cf9203f5ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a41ad0-8665dcdf7d-948h2" Nov 03 21:01:00.617535 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.617501 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5611d11d-6bc4-4725-a5e7-d2cf9203f5ce-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a41ad0-8665dcdf7d-948h2\" (UID: \"5611d11d-6bc4-4725-a5e7-d2cf9203f5ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a41ad0-8665dcdf7d-948h2" Nov 03 21:01:00.619365 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.619333 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce51a38f-47a9-4926-adec-cecfceba6000-maas-api-tls\") pod \"maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr\" (UID: \"ce51a38f-47a9-4926-adec-cecfceba6000\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr" Nov 03 21:01:00.675664 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.675636 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v55ss\" (UniqueName: \"kubernetes.io/projected/5611d11d-6bc4-4725-a5e7-d2cf9203f5ce-kube-api-access-v55ss\") pod \"payload-processing-e2e-worker-w1-a41ad0-8665dcdf7d-948h2\" (UID: \"5611d11d-6bc4-4725-a5e7-d2cf9203f5ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a41ad0-8665dcdf7d-948h2" Nov 03 21:01:00.826835 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.826790 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr" Nov 03 21:01:00.828067 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.826897 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a41ad0-8665dcdf7d-948h2" Nov 03 21:01:00.970219 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:00.970170 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a41ad0-5458b6d47-62h44"] Nov 03 21:01:00.971751 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:01:00.971559 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38461555_612e_478c_a9f8_98807b526d0f.slice/crio-4eeeff456f0eb0c6316a773a93a50c018426ad32350b51cf072a41dc1294f6b7 WatchSource:0}: Error finding container 4eeeff456f0eb0c6316a773a93a50c018426ad32350b51cf072a41dc1294f6b7: Status 404 returned error can't find the container with id 4eeeff456f0eb0c6316a773a93a50c018426ad32350b51cf072a41dc1294f6b7 Nov 03 21:01:01.206345 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:01.206318 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a41ad0-8665dcdf7d-948h2"] Nov 03 21:01:01.208056 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:01:01.208027 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5611d11d_6bc4_4725_a5e7_d2cf9203f5ce.slice/crio-665b39d1c295abd581b016aedffb80288665515f41de56dd518a2eeb21f25791 WatchSource:0}: Error finding container 665b39d1c295abd581b016aedffb80288665515f41de56dd518a2eeb21f25791: Status 404 returned error can't find the container with id 665b39d1c295abd581b016aedffb80288665515f41de56dd518a2eeb21f25791 Nov 03 21:01:01.209732 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:01.209701 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a41ad0-5458b6d47-62h44" event={"ID":"38461555-612e-478c-a9f8-98807b526d0f","Type":"ContainerStarted","Data":"4c5111ca72ef0cf130b501a8f3884341c22777339e3b4b858b7c12c21f93c220"} Nov 03 21:01:01.209893 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:01.209763 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a41ad0-5458b6d47-62h44" event={"ID":"38461555-612e-478c-a9f8-98807b526d0f","Type":"ContainerStarted","Data":"4eeeff456f0eb0c6316a773a93a50c018426ad32350b51cf072a41dc1294f6b7"} Nov 03 21:01:01.209979 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:01.209911 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a41ad0-5458b6d47-62h44" Nov 03 21:01:01.213665 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:01.211988 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr"] Nov 03 21:01:01.213665 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:01.212954 5615 generic.go:356] "Generic (PLEG): container finished" podID="2a10a87e-4975-45cd-9f46-d9246dd3ae40" containerID="f986576b349e1b26761aab07bd273735ee57328db214e15ca51196de238ca8d7" exitCode=0 Nov 03 21:01:01.213665 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:01.212991 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" event={"ID":"2a10a87e-4975-45cd-9f46-d9246dd3ae40","Type":"ContainerDied","Data":"f986576b349e1b26761aab07bd273735ee57328db214e15ca51196de238ca8d7"} Nov 03 21:01:01.213665 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:01.213422 5615 scope.go:122] "RemoveContainer" containerID="f986576b349e1b26761aab07bd273735ee57328db214e15ca51196de238ca8d7" Nov 03 21:01:01.214769 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:01.214739 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:01:01.214883 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:01.214819 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:01:01.214883 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:01:01.214864 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce51a38f_47a9_4926_adec_cecfceba6000.slice/crio-c3c0ed092d705203ba81286266ab4599dd561db7c705146892d5b7d4f4ad99f6 WatchSource:0}: Error finding container c3c0ed092d705203ba81286266ab4599dd561db7c705146892d5b7d4f4ad99f6: Status 404 returned error can't find the container with id c3c0ed092d705203ba81286266ab4599dd561db7c705146892d5b7d4f4ad99f6 Nov 03 21:01:01.214976 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:01.214893 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:01:01.320923 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:01.320866 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a41ad0-5458b6d47-62h44" podStartSLOduration=1.320851147 podStartE2EDuration="1.320851147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:01.318949571 +0000 UTC m=+755.786728298" watchObservedRunningTime="2026-11-03 21:01:01.320851147 +0000 UTC m=+755.788629907" Nov 03 21:01:02.235158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:02.235123 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr" event={"ID":"ce51a38f-47a9-4926-adec-cecfceba6000","Type":"ContainerStarted","Data":"c3c0ed092d705203ba81286266ab4599dd561db7c705146892d5b7d4f4ad99f6"} Nov 03 21:01:02.236956 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:02.236926 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a41ad0-8665dcdf7d-948h2" event={"ID":"5611d11d-6bc4-4725-a5e7-d2cf9203f5ce","Type":"ContainerStarted","Data":"81508a73df357b42ccd54bcdfa173bb0298ff9da6234fe73ea1d7cbafe90625f"} Nov 03 21:01:02.237087 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:02.236985 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a41ad0-8665dcdf7d-948h2" event={"ID":"5611d11d-6bc4-4725-a5e7-d2cf9203f5ce","Type":"ContainerStarted","Data":"665b39d1c295abd581b016aedffb80288665515f41de56dd518a2eeb21f25791"} Nov 03 21:01:02.237150 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:02.237097 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-a41ad0-8665dcdf7d-948h2" Nov 03 21:01:02.239036 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:02.239009 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" event={"ID":"2a10a87e-4975-45cd-9f46-d9246dd3ae40","Type":"ContainerStarted","Data":"8ece380fe7f136457a470696900c7f1c7f159db587787bdc95416f236cf5c149"} Nov 03 21:01:02.289426 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:02.289394 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:01:02.289615 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:02.289594 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 03 21:01:02.289664 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:02.289646 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" podUID="2a10a87e-4975-45cd-9f46-d9246dd3ae40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 03 21:01:02.346153 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:02.346097 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-a41ad0-8665dcdf7d-948h2" podStartSLOduration=2.346065547 podStartE2EDuration="2.346065547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:02.345618535 +0000 UTC m=+756.813397259" watchObservedRunningTime="2026-11-03 21:01:02.346065547 +0000 UTC m=+756.813844270" Nov 03 21:01:03.245839 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:03.245798 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr" event={"ID":"ce51a38f-47a9-4926-adec-cecfceba6000","Type":"ContainerStarted","Data":"831ac3c020c9166e642c67066767332bd165c1f91e62b25a0231bde74d233927"} Nov 03 21:01:03.246366 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:03.246221 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr" Nov 03 21:01:03.289362 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:03.289323 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 03 21:01:03.289517 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:03.289381 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" podUID="2a10a87e-4975-45cd-9f46-d9246dd3ae40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 03 21:01:03.358519 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:03.358466 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr" podStartSLOduration=3.384363815 podStartE2EDuration="4.358449831s" totalImagesPullingTime="974.086016ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:03.356864182 +0000 UTC m=+757.824642908" watchObservedRunningTime="2026-11-03 21:01:03.358449831 +0000 UTC m=+757.826228545" Nov 03 21:01:04.289530 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:04.289491 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 03 21:01:04.289958 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:04.289583 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" podUID="2a10a87e-4975-45cd-9f46-d9246dd3ae40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 03 21:01:05.288949 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:05.288910 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 03 21:01:05.289176 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:05.288966 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" podUID="2a10a87e-4975-45cd-9f46-d9246dd3ae40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 03 21:01:06.289251 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:06.289210 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 03 21:01:06.289645 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:06.289293 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" podUID="2a10a87e-4975-45cd-9f46-d9246dd3ae40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 03 21:01:07.289223 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:07.289172 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 03 21:01:07.289435 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:07.289247 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" podUID="2a10a87e-4975-45cd-9f46-d9246dd3ae40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 03 21:01:08.292622 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:08.292582 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:01:08.293142 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:08.293005 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:01:08.293865 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:08.293841 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:01:09.258271 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:09.258244 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr" Nov 03 21:01:10.641135 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:01:10.641100 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:01:12.242439 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:12.242404 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a41ad0-5458b6d47-62h44" Nov 03 21:01:13.249241 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:13.249212 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-a41ad0-8665dcdf7d-948h2" Nov 03 21:01:17.320263 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:17.320227 5615 generic.go:356] "Generic (PLEG): container finished" podID="884c8308-cf0b-496d-9f55-bb6563fe0238" containerID="46727b5c84e14613cb6fd6dbdc300439f0b7bba72f858cbd7927e91592837e79" exitCode=0 Nov 03 21:01:17.320705 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:17.320276 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" event={"ID":"884c8308-cf0b-496d-9f55-bb6563fe0238","Type":"ContainerDied","Data":"46727b5c84e14613cb6fd6dbdc300439f0b7bba72f858cbd7927e91592837e79"} Nov 03 21:01:17.320705 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:17.320651 5615 scope.go:122] "RemoveContainer" containerID="46727b5c84e14613cb6fd6dbdc300439f0b7bba72f858cbd7927e91592837e79" Nov 03 21:01:17.321910 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:17.321875 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:01:17.321966 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:17.321951 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:01:17.322006 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:17.321982 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:01:18.334490 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:18.334453 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" event={"ID":"884c8308-cf0b-496d-9f55-bb6563fe0238","Type":"ContainerStarted","Data":"7d264a294afeecb09d04c751e3d95b241f79f5d3530f1e6860b948c2ddf9783c"} Nov 03 21:01:19.028995 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:19.028963 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:01:19.141690 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:19.141652 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:01:19.142773 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:19.142749 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 21:01:19.142879 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:19.142807 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" podUID="884c8308-cf0b-496d-9f55-bb6563fe0238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 21:01:20.141557 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:20.141517 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 21:01:20.141982 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:20.141577 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" podUID="884c8308-cf0b-496d-9f55-bb6563fe0238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 21:01:21.141969 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:21.141934 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 21:01:21.142413 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:21.141997 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" podUID="884c8308-cf0b-496d-9f55-bb6563fe0238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 21:01:22.141772 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:22.141734 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 21:01:22.141963 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:22.141800 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" podUID="884c8308-cf0b-496d-9f55-bb6563fe0238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 21:01:23.142452 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:23.142417 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 21:01:23.142870 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:23.142472 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" podUID="884c8308-cf0b-496d-9f55-bb6563fe0238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 21:01:24.146634 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:24.146600 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:01:24.147704 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:01:24.147683 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:01:40.618461 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:01:40.618419 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:02:10.618738 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:02:10.618661 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:02:35.183855 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:35.183820 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz"] Nov 03 21:02:35.187640 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:35.187615 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" Nov 03 21:02:35.250401 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:35.250364 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a89b0627-f993-4003-ac8b-fc449b44d9d9-dshm\") pod \"simulator-w1-a41ad0-kserve-6d57df497-n2cjz\" (UID: \"a89b0627-f993-4003-ac8b-fc449b44d9d9\") " pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" Nov 03 21:02:35.250566 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:35.250436 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a89b0627-f993-4003-ac8b-fc449b44d9d9-tls-certs\") pod \"simulator-w1-a41ad0-kserve-6d57df497-n2cjz\" (UID: \"a89b0627-f993-4003-ac8b-fc449b44d9d9\") " pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" Nov 03 21:02:35.250614 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:35.250573 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a89b0627-f993-4003-ac8b-fc449b44d9d9-model-cache\") pod \"simulator-w1-a41ad0-kserve-6d57df497-n2cjz\" (UID: \"a89b0627-f993-4003-ac8b-fc449b44d9d9\") " pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" Nov 03 21:02:35.250675 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:35.250611 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a89b0627-f993-4003-ac8b-fc449b44d9d9-tmp-dir\") pod \"simulator-w1-a41ad0-kserve-6d57df497-n2cjz\" (UID: \"a89b0627-f993-4003-ac8b-fc449b44d9d9\") " pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" Nov 03 21:02:35.250675 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:35.250639 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a89b0627-f993-4003-ac8b-fc449b44d9d9-home\") pod \"simulator-w1-a41ad0-kserve-6d57df497-n2cjz\" (UID: \"a89b0627-f993-4003-ac8b-fc449b44d9d9\") " pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" Nov 03 21:02:35.250790 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:35.250762 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz2tz\" (UniqueName: \"kubernetes.io/projected/a89b0627-f993-4003-ac8b-fc449b44d9d9-kube-api-access-cz2tz\") pod \"simulator-w1-a41ad0-kserve-6d57df497-n2cjz\" (UID: \"a89b0627-f993-4003-ac8b-fc449b44d9d9\") " pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" Nov 03 21:02:35.284689 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:35.284655 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a41ad0\"/\"default-dockercfg-zqb9q\"" Nov 03 21:02:35.284689 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:35.284660 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a41ad0\"/\"simulator-w1-a41ad0-kserve-self-signed-certs\"" Nov 03 21:02:35.284907 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:35.284699 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a41ad0\"/\"openshift-service-ca.crt\"" Nov 03 21:02:35.284907 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:35.284704 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a41ad0\"/\"kube-root-ca.crt\"" Nov 03 21:02:35.286611 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:35.286590 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz"] Nov 03 21:02:35.352394 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:35.352346 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a89b0627-f993-4003-ac8b-fc449b44d9d9-model-cache\") pod \"simulator-w1-a41ad0-kserve-6d57df497-n2cjz\" (UID: \"a89b0627-f993-4003-ac8b-fc449b44d9d9\") " pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" Nov 03 21:02:35.352571 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:35.352407 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a89b0627-f993-4003-ac8b-fc449b44d9d9-tmp-dir\") pod \"simulator-w1-a41ad0-kserve-6d57df497-n2cjz\" (UID: \"a89b0627-f993-4003-ac8b-fc449b44d9d9\") " pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" Nov 03 21:02:35.352571 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:35.352451 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a89b0627-f993-4003-ac8b-fc449b44d9d9-home\") pod \"simulator-w1-a41ad0-kserve-6d57df497-n2cjz\" (UID: \"a89b0627-f993-4003-ac8b-fc449b44d9d9\") " pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" Nov 03 21:02:35.352571 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:35.352510 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz2tz\" (UniqueName: \"kubernetes.io/projected/a89b0627-f993-4003-ac8b-fc449b44d9d9-kube-api-access-cz2tz\") pod \"simulator-w1-a41ad0-kserve-6d57df497-n2cjz\" (UID: \"a89b0627-f993-4003-ac8b-fc449b44d9d9\") " pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" Nov 03 21:02:35.352571 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:35.352547 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a89b0627-f993-4003-ac8b-fc449b44d9d9-dshm\") pod \"simulator-w1-a41ad0-kserve-6d57df497-n2cjz\" (UID: \"a89b0627-f993-4003-ac8b-fc449b44d9d9\") " pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" Nov 03 21:02:35.352778 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:35.352630 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a89b0627-f993-4003-ac8b-fc449b44d9d9-tls-certs\") pod \"simulator-w1-a41ad0-kserve-6d57df497-n2cjz\" (UID: \"a89b0627-f993-4003-ac8b-fc449b44d9d9\") " pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" Nov 03 21:02:35.352831 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:35.352802 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a89b0627-f993-4003-ac8b-fc449b44d9d9-tmp-dir\") pod \"simulator-w1-a41ad0-kserve-6d57df497-n2cjz\" (UID: \"a89b0627-f993-4003-ac8b-fc449b44d9d9\") " pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" Nov 03 21:02:35.352889 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:35.352867 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a89b0627-f993-4003-ac8b-fc449b44d9d9-home\") pod \"simulator-w1-a41ad0-kserve-6d57df497-n2cjz\" (UID: \"a89b0627-f993-4003-ac8b-fc449b44d9d9\") " pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" Nov 03 21:02:35.352955 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:35.352904 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a89b0627-f993-4003-ac8b-fc449b44d9d9-model-cache\") pod \"simulator-w1-a41ad0-kserve-6d57df497-n2cjz\" (UID: \"a89b0627-f993-4003-ac8b-fc449b44d9d9\") " pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" Nov 03 21:02:35.354859 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:35.354833 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a89b0627-f993-4003-ac8b-fc449b44d9d9-dshm\") pod \"simulator-w1-a41ad0-kserve-6d57df497-n2cjz\" (UID: \"a89b0627-f993-4003-ac8b-fc449b44d9d9\") " pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" Nov 03 21:02:35.355044 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:35.355028 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a89b0627-f993-4003-ac8b-fc449b44d9d9-tls-certs\") pod \"simulator-w1-a41ad0-kserve-6d57df497-n2cjz\" (UID: \"a89b0627-f993-4003-ac8b-fc449b44d9d9\") " pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" Nov 03 21:02:35.414607 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:35.414566 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz2tz\" (UniqueName: \"kubernetes.io/projected/a89b0627-f993-4003-ac8b-fc449b44d9d9-kube-api-access-cz2tz\") pod \"simulator-w1-a41ad0-kserve-6d57df497-n2cjz\" (UID: \"a89b0627-f993-4003-ac8b-fc449b44d9d9\") " pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" Nov 03 21:02:35.499674 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:35.499633 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" Nov 03 21:02:35.882734 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:35.882705 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz"] Nov 03 21:02:35.883593 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:02:35.883565 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda89b0627_f993_4003_ac8b_fc449b44d9d9.slice/crio-1c6e6ba78c400559800abda8c9c553a5156a04e3b17e2f0b0c41bd76712c8ea5 WatchSource:0}: Error finding container 1c6e6ba78c400559800abda8c9c553a5156a04e3b17e2f0b0c41bd76712c8ea5: Status 404 returned error can't find the container with id 1c6e6ba78c400559800abda8c9c553a5156a04e3b17e2f0b0c41bd76712c8ea5 Nov 03 21:02:36.740745 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:36.740708 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" event={"ID":"a89b0627-f993-4003-ac8b-fc449b44d9d9","Type":"ContainerStarted","Data":"cd9400e70c37c0078b7bc57f940c1448918fd6bd6f7a48c9902fdd2145238309"} Nov 03 21:02:36.740745 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:36.740750 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" event={"ID":"a89b0627-f993-4003-ac8b-fc449b44d9d9","Type":"ContainerStarted","Data":"1c6e6ba78c400559800abda8c9c553a5156a04e3b17e2f0b0c41bd76712c8ea5"} Nov 03 21:02:36.852242 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:36.852187 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" podStartSLOduration=1.8521713659999999 podStartE2EDuration="1.852171366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:02:36.851664561 +0000 UTC m=+851.319443286" watchObservedRunningTime="2026-11-03 21:02:36.852171366 +0000 UTC m=+851.319950088" Nov 03 21:02:37.500790 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:37.500754 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" Nov 03 21:02:40.618721 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:02:40.618674 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:02:45.500376 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:45.500333 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" Nov 03 21:02:45.514259 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:45.514230 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" Nov 03 21:02:45.801270 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:02:45.801187 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" Nov 03 21:03:10.618842 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:03:10.618805 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:03:40.619203 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:03:40.619119 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:04:01.277301 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.277263 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9"] Nov 03 21:04:01.280469 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.280450 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:01.295749 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.295720 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:01.295889 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.295755 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:01.295889 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.295781 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfbrn\" (UniqueName: \"kubernetes.io/projected/13f7f450-acbe-4a27-aedc-4c3f43c942e5-kube-api-access-qfbrn\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:01.295996 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.295894 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:01.295996 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.295952 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:01.296109 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.296026 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:01.296109 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.296055 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:01.296227 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.296162 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:01.296227 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.296192 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:01.330484 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.330449 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-099a5e-openshift-default-dockercfg-2h6nd\"" Nov 03 21:04:01.379840 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.379803 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9"] Nov 03 21:04:01.397028 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.396998 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:01.397028 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.397033 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:01.397304 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.397054 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:01.397304 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.397117 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:01.397304 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.397132 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:01.397304 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.397239 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:01.397304 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.397267 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:01.397304 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.397291 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfbrn\" (UniqueName: \"kubernetes.io/projected/13f7f450-acbe-4a27-aedc-4c3f43c942e5-kube-api-access-qfbrn\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:01.397631 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.397330 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:01.397631 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.397497 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:01.397631 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.397564 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:01.397631 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.397622 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:01.397922 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.397896 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:01.398002 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.397984 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:01.399803 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.399776 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:01.400169 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.400146 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:01.456781 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.456615 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfbrn\" (UniqueName: \"kubernetes.io/projected/13f7f450-acbe-4a27-aedc-4c3f43c942e5-kube-api-access-qfbrn\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:01.457418 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.457386 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:01.592479 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:01.592392 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:02.008212 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:02.008187 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9"] Nov 03 21:04:02.010085 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:04:02.010041 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f7f450_acbe_4a27_aedc_4c3f43c942e5.slice/crio-20a176a57d98fe83ec15c1b47a7f1f2ad89786d6a710e0da3bb360109fe09798 WatchSource:0}: Error finding container 20a176a57d98fe83ec15c1b47a7f1f2ad89786d6a710e0da3bb360109fe09798: Status 404 returned error can't find the container with id 20a176a57d98fe83ec15c1b47a7f1f2ad89786d6a710e0da3bb360109fe09798 Nov 03 21:04:02.012429 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:02.012393 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:04:02.012526 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:02.012463 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:04:02.012526 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:02.012491 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:04:02.207722 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:02.207688 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" event={"ID":"13f7f450-acbe-4a27-aedc-4c3f43c942e5","Type":"ContainerStarted","Data":"fa9b3391393f8f5ba0aa10203e9226dbb3562ff89fc814bd17b402d9aa6da0ff"} Nov 03 21:04:02.207870 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:02.207742 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" event={"ID":"13f7f450-acbe-4a27-aedc-4c3f43c942e5","Type":"ContainerStarted","Data":"20a176a57d98fe83ec15c1b47a7f1f2ad89786d6a710e0da3bb360109fe09798"} Nov 03 21:04:02.324561 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:02.324452 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" podStartSLOduration=1.324429412 podStartE2EDuration="1.324429412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:02.321988255 +0000 UTC m=+936.789767011" watchObservedRunningTime="2026-11-03 21:04:02.324429412 +0000 UTC m=+936.792208138" Nov 03 21:04:02.592718 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:02.592586 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:03.594638 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:03.594597 5615 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 21:04:03.595161 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:03.594682 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" podUID="13f7f450-acbe-4a27-aedc-4c3f43c942e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 21:04:04.593657 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:04.593621 5615 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 21:04:04.593839 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:04.593672 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" podUID="13f7f450-acbe-4a27-aedc-4c3f43c942e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 21:04:05.592730 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:05.592694 5615 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 21:04:05.593141 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:05.592755 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" podUID="13f7f450-acbe-4a27-aedc-4c3f43c942e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 21:04:06.593736 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:06.593689 5615 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 21:04:06.594295 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:06.593747 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" podUID="13f7f450-acbe-4a27-aedc-4c3f43c942e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 21:04:07.593040 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:07.593001 5615 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 21:04:07.593326 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:07.593059 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" podUID="13f7f450-acbe-4a27-aedc-4c3f43c942e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 21:04:09.418609 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:09.418577 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:09.419157 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:09.419112 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:09.423624 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:09.423601 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:04:10.618552 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:04:10.618511 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:04:20.213874 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.213840 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t"] Nov 03 21:04:20.217040 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.217016 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:20.266567 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.266529 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a41ad0-openshift-default-dockercfg-dhkbv\"" Nov 03 21:04:20.318778 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.318738 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t"] Nov 03 21:04:20.322343 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.322299 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:20.322529 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.322351 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:20.322529 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.322462 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:20.322649 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.322536 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:20.322649 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.322604 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgjwk\" (UniqueName: \"kubernetes.io/projected/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-kube-api-access-xgjwk\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:20.322761 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.322676 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:20.322761 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.322717 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:20.322876 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.322787 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:20.322935 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.322895 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:20.424408 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.424355 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:20.424629 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.424438 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:20.424629 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.424460 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:20.424629 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.424482 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:20.424629 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.424509 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:20.424852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.424682 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgjwk\" (UniqueName: \"kubernetes.io/projected/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-kube-api-access-xgjwk\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:20.424852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.424728 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:20.424852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.424779 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:20.424852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.424828 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:20.425060 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.424960 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:20.425140 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.425093 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:20.425418 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.425391 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:20.425510 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.425413 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:20.425510 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.425464 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:20.427240 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.427216 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:20.427469 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.427450 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:20.482532 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.482502 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:20.482719 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.482695 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgjwk\" (UniqueName: \"kubernetes.io/projected/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-kube-api-access-xgjwk\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:20.530638 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.530582 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:20.946388 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.946358 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t"] Nov 03 21:04:20.948207 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:04:20.948179 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d42f7c_c6eb_4ab3_a1d2_9d8a7572050b.slice/crio-9d03e5156d564f87d89c147d1aa62eeb42714eabcd5c33e39229e37e7601efa8 WatchSource:0}: Error finding container 9d03e5156d564f87d89c147d1aa62eeb42714eabcd5c33e39229e37e7601efa8: Status 404 returned error can't find the container with id 9d03e5156d564f87d89c147d1aa62eeb42714eabcd5c33e39229e37e7601efa8 Nov 03 21:04:20.950743 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.950705 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:04:20.950827 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.950780 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:04:20.950827 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:20.950817 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:04:21.347025 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:21.346989 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" event={"ID":"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b","Type":"ContainerStarted","Data":"1e92f3208be893b817dd50dbc19bc41713288b24dbb2c2a80fd7c5a495e63e94"} Nov 03 21:04:21.347445 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:21.347039 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" event={"ID":"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b","Type":"ContainerStarted","Data":"9d03e5156d564f87d89c147d1aa62eeb42714eabcd5c33e39229e37e7601efa8"} Nov 03 21:04:21.462659 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:21.462605 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" podStartSLOduration=1.462592822 podStartE2EDuration="1.462592822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:21.461852372 +0000 UTC m=+955.929631096" watchObservedRunningTime="2026-11-03 21:04:21.462592822 +0000 UTC m=+955.930371544" Nov 03 21:04:21.530984 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:21.530947 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:22.532619 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:22.532585 5615 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 21:04:22.532990 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:22.532642 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" podUID="a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 21:04:23.531581 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:23.531549 5615 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 21:04:23.531756 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:23.531604 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" podUID="a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 21:04:24.532679 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:24.532636 5615 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 21:04:24.533302 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:24.533270 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" podUID="a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 21:04:25.386236 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:25.386202 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:04:25.386484 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:25.386465 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" podUID="889d1dcf-c570-4151-9884-ddc9962a4c0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:04:25.535497 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:25.535457 5615 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 21:04:25.536132 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:25.536098 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" podUID="a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 21:04:26.531486 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:26.531447 5615 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 21:04:26.531700 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:26.531506 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" podUID="a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 21:04:27.879027 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:27.878987 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:27.879595 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:27.879573 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:27.880503 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:27.880480 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 21:04:40.618150 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:04:40.618112 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:04:44.066276 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.066237 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw"] Nov 03 21:04:44.069883 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.069855 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:44.154460 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.154418 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-l9qgw\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:44.154660 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.154612 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-l9qgw\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:44.154744 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.154667 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-l9qgw\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:44.154744 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.154702 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-l9qgw\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:44.154744 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.154735 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-l9qgw\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:44.154885 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.154785 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-l9qgw\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:44.154885 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.154817 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-l9qgw\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:44.154885 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.154858 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64lp2\" (UniqueName: \"kubernetes.io/projected/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-kube-api-access-64lp2\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-l9qgw\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:44.154979 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.154925 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-l9qgw\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:44.169839 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.169802 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw"] Nov 03 21:04:44.256090 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.256036 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-l9qgw\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:44.256303 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.256115 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-l9qgw\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:44.256303 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.256141 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-l9qgw\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:44.256303 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.256162 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-l9qgw\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:44.256303 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.256195 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-l9qgw\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:44.256303 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.256218 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-l9qgw\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:44.256303 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.256257 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64lp2\" (UniqueName: \"kubernetes.io/projected/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-kube-api-access-64lp2\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-l9qgw\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:44.256653 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.256317 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-l9qgw\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:44.256653 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.256378 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-l9qgw\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:44.256791 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.256770 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-l9qgw\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:44.257035 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.256995 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-l9qgw\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:44.257458 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.257429 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-l9qgw\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:44.257615 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.257588 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-l9qgw\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:44.258327 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.258300 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-l9qgw\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:44.259777 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.259746 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-l9qgw\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:44.260568 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.260545 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-l9qgw\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:44.316866 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.316658 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64lp2\" (UniqueName: \"kubernetes.io/projected/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-kube-api-access-64lp2\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-l9qgw\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:44.316866 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.316674 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-l9qgw\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:44.383237 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.383202 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:44.809138 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.808962 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw"] Nov 03 21:04:44.815930 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.815884 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:04:44.816048 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.815980 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:04:44.816048 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:44.816027 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:04:45.495313 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:45.495274 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" event={"ID":"ebbedd3d-03f0-4518-b13b-fd88e4c633d5","Type":"ContainerStarted","Data":"457218192f6b6671c149e4065085b665ee543326e0909e3a0990fa832e93ebee"} Nov 03 21:04:45.495712 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:45.495333 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" event={"ID":"ebbedd3d-03f0-4518-b13b-fd88e4c633d5","Type":"ContainerStarted","Data":"e24bb392976f91973b5f0b86a0cc071587c55c89f5cab18c17e803ce3adc0c10"} Nov 03 21:04:45.610412 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:45.610347 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" podStartSLOduration=1.610326783 podStartE2EDuration="1.610326783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:45.609648068 +0000 UTC m=+980.077426791" watchObservedRunningTime="2026-11-03 21:04:45.610326783 +0000 UTC m=+980.078105507" Nov 03 21:04:46.383735 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:46.383685 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:46.385598 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:46.385572 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 21:04:46.385748 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:46.385633 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" podUID="ebbedd3d-03f0-4518-b13b-fd88e4c633d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 21:04:47.384716 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:47.384670 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 21:04:47.385135 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:47.384765 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" podUID="ebbedd3d-03f0-4518-b13b-fd88e4c633d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 21:04:48.383821 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:48.383781 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 21:04:48.384012 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:48.383844 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" podUID="ebbedd3d-03f0-4518-b13b-fd88e4c633d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 21:04:49.225252 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:49.225208 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s"] Nov 03 21:04:49.228901 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:49.228869 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s" Nov 03 21:04:49.326687 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:49.326489 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-254f80d1d482\"" Nov 03 21:04:49.326687 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:49.326503 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 21:04:49.326687 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:49.326503 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 21:04:49.326687 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:49.326602 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 21:04:49.326687 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:49.326650 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 21:04:49.326687 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:49.326651 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-m9wt4\"" Nov 03 21:04:49.329811 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:49.329777 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s"] Nov 03 21:04:49.330419 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:49.330391 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6df74952-1d07-4288-8ea3-d844cb913fc8-provider-credentials\") pod \"payload-processing-external-model-56cc46bb7d-58w9s\" (UID: \"6df74952-1d07-4288-8ea3-d844cb913fc8\") " pod="models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s" Nov 03 21:04:49.330518 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:49.330440 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6df74952-1d07-4288-8ea3-d844cb913fc8-routing-overlay-volume\") pod \"payload-processing-external-model-56cc46bb7d-58w9s\" (UID: \"6df74952-1d07-4288-8ea3-d844cb913fc8\") " pod="models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s" Nov 03 21:04:49.330587 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:49.330538 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df74952-1d07-4288-8ea3-d844cb913fc8-plugins-config-volume\") pod \"payload-processing-external-model-56cc46bb7d-58w9s\" (UID: \"6df74952-1d07-4288-8ea3-d844cb913fc8\") " pod="models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s" Nov 03 21:04:49.330645 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:49.330587 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v92lv\" (UniqueName: \"kubernetes.io/projected/6df74952-1d07-4288-8ea3-d844cb913fc8-kube-api-access-v92lv\") pod \"payload-processing-external-model-56cc46bb7d-58w9s\" (UID: \"6df74952-1d07-4288-8ea3-d844cb913fc8\") " pod="models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s" Nov 03 21:04:49.383768 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:49.383731 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 21:04:49.383957 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:49.383784 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" podUID="ebbedd3d-03f0-4518-b13b-fd88e4c633d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 21:04:49.431556 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:49.431519 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df74952-1d07-4288-8ea3-d844cb913fc8-plugins-config-volume\") pod \"payload-processing-external-model-56cc46bb7d-58w9s\" (UID: \"6df74952-1d07-4288-8ea3-d844cb913fc8\") " pod="models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s" Nov 03 21:04:49.431742 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:49.431585 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v92lv\" (UniqueName: \"kubernetes.io/projected/6df74952-1d07-4288-8ea3-d844cb913fc8-kube-api-access-v92lv\") pod \"payload-processing-external-model-56cc46bb7d-58w9s\" (UID: \"6df74952-1d07-4288-8ea3-d844cb913fc8\") " pod="models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s" Nov 03 21:04:49.431742 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:49.431680 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6df74952-1d07-4288-8ea3-d844cb913fc8-provider-credentials\") pod \"payload-processing-external-model-56cc46bb7d-58w9s\" (UID: \"6df74952-1d07-4288-8ea3-d844cb913fc8\") " pod="models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s" Nov 03 21:04:49.431742 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:49.431707 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6df74952-1d07-4288-8ea3-d844cb913fc8-routing-overlay-volume\") pod \"payload-processing-external-model-56cc46bb7d-58w9s\" (UID: \"6df74952-1d07-4288-8ea3-d844cb913fc8\") " pod="models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s" Nov 03 21:04:49.432336 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:49.432312 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6df74952-1d07-4288-8ea3-d844cb913fc8-routing-overlay-volume\") pod \"payload-processing-external-model-56cc46bb7d-58w9s\" (UID: \"6df74952-1d07-4288-8ea3-d844cb913fc8\") " pod="models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s" Nov 03 21:04:49.432453 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:49.432365 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df74952-1d07-4288-8ea3-d844cb913fc8-plugins-config-volume\") pod \"payload-processing-external-model-56cc46bb7d-58w9s\" (UID: \"6df74952-1d07-4288-8ea3-d844cb913fc8\") " pod="models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s" Nov 03 21:04:49.434195 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:49.434164 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6df74952-1d07-4288-8ea3-d844cb913fc8-provider-credentials\") pod \"payload-processing-external-model-56cc46bb7d-58w9s\" (UID: \"6df74952-1d07-4288-8ea3-d844cb913fc8\") " pod="models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s" Nov 03 21:04:49.493338 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:49.493304 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v92lv\" (UniqueName: \"kubernetes.io/projected/6df74952-1d07-4288-8ea3-d844cb913fc8-kube-api-access-v92lv\") pod \"payload-processing-external-model-56cc46bb7d-58w9s\" (UID: \"6df74952-1d07-4288-8ea3-d844cb913fc8\") " pod="models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s" Nov 03 21:04:49.542506 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:49.542463 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s" Nov 03 21:04:49.923481 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:49.923456 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s"] Nov 03 21:04:49.927589 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:04:49.927558 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df74952_1d07_4288_8ea3_d844cb913fc8.slice/crio-a2e038eb5f44d9c52d3e5d1823c0dec3c6b0a8bc025c1a9194c26492fd8cd5aa WatchSource:0}: Error finding container a2e038eb5f44d9c52d3e5d1823c0dec3c6b0a8bc025c1a9194c26492fd8cd5aa: Status 404 returned error can't find the container with id a2e038eb5f44d9c52d3e5d1823c0dec3c6b0a8bc025c1a9194c26492fd8cd5aa Nov 03 21:04:50.383806 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:50.383764 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 21:04:50.384256 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:50.383852 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" podUID="ebbedd3d-03f0-4518-b13b-fd88e4c633d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 21:04:50.546698 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:50.546655 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s" event={"ID":"6df74952-1d07-4288-8ea3-d844cb913fc8","Type":"ContainerStarted","Data":"58a5027d0d849478376e1ebed780da452366847434eb66d3576724df71696aff"} Nov 03 21:04:50.546898 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:50.546708 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s" event={"ID":"6df74952-1d07-4288-8ea3-d844cb913fc8","Type":"ContainerStarted","Data":"a2e038eb5f44d9c52d3e5d1823c0dec3c6b0a8bc025c1a9194c26492fd8cd5aa"} Nov 03 21:04:50.546898 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:50.546794 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s" Nov 03 21:04:50.658260 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:50.658167 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s" podStartSLOduration=1.6581489139999999 podStartE2EDuration="1.658148914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:50.656887182 +0000 UTC m=+985.124665931" watchObservedRunningTime="2026-11-03 21:04:50.658148914 +0000 UTC m=+985.125927638" Nov 03 21:04:51.384469 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:51.384420 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 21:04:51.385001 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:51.384478 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" podUID="ebbedd3d-03f0-4518-b13b-fd88e4c633d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 21:04:52.384490 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:52.384449 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 21:04:52.384891 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:52.384517 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" podUID="ebbedd3d-03f0-4518-b13b-fd88e4c633d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 21:04:53.744167 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:53.744132 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:53.746760 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:53.746723 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:53.747713 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:53.747689 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:04:59.202288 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.202250 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr"] Nov 03 21:04:59.205490 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.205466 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:04:59.248478 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.248438 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hzgjr\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:04:59.248875 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.248849 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a14fb4-c56f-4704-91c9-bd2de3648f55-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hzgjr\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:04:59.249222 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.249177 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hzgjr\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:04:59.249418 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.249398 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a14fb4-c56f-4704-91c9-bd2de3648f55-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hzgjr\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:04:59.249582 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.249566 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hzgjr\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:04:59.249812 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.249792 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a14fb4-c56f-4704-91c9-bd2de3648f55-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hzgjr\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:04:59.249967 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.249952 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hzgjr\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:04:59.250137 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.250120 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hzgjr\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:04:59.250315 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.250300 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgt9n\" (UniqueName: \"kubernetes.io/projected/81a14fb4-c56f-4704-91c9-bd2de3648f55-kube-api-access-wgt9n\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hzgjr\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:04:59.255375 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.255347 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2"] Nov 03 21:04:59.259466 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.259448 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:04:59.354168 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.351794 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hzgjr\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:04:59.354168 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.351853 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-msdt2\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:04:59.354168 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.351881 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-msdt2\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:04:59.354168 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.351908 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a14fb4-c56f-4704-91c9-bd2de3648f55-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hzgjr\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:04:59.354168 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.351941 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-msdt2\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:04:59.354168 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.351989 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-msdt2\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:04:59.354168 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.352014 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbmzs\" (UniqueName: \"kubernetes.io/projected/e3271312-1e59-4a84-8ac1-d8e57f69f85b-kube-api-access-zbmzs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-msdt2\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:04:59.354168 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.352050 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hzgjr\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:04:59.354168 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.352086 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a14fb4-c56f-4704-91c9-bd2de3648f55-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hzgjr\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:04:59.354168 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.352111 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hzgjr\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:04:59.354168 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.352142 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-msdt2\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:04:59.354168 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.352180 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-msdt2\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:04:59.354168 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.352217 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a14fb4-c56f-4704-91c9-bd2de3648f55-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hzgjr\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:04:59.354168 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.352240 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hzgjr\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:04:59.354168 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.352266 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hzgjr\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:04:59.354168 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.352286 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-msdt2\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:04:59.354168 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.352344 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgt9n\" (UniqueName: \"kubernetes.io/projected/81a14fb4-c56f-4704-91c9-bd2de3648f55-kube-api-access-wgt9n\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hzgjr\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:04:59.354168 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.352411 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-msdt2\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:04:59.354168 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.352854 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hzgjr\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:04:59.354168 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.353478 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a14fb4-c56f-4704-91c9-bd2de3648f55-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hzgjr\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:04:59.354168 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.353747 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hzgjr\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:04:59.356116 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.355996 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hzgjr\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:04:59.356909 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.356622 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hzgjr\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:04:59.380618 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.377982 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a14fb4-c56f-4704-91c9-bd2de3648f55-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hzgjr\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:04:59.380618 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.378248 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hzgjr\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:04:59.389042 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.389008 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2"] Nov 03 21:04:59.414940 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.414840 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr"] Nov 03 21:04:59.418781 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.418095 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a14fb4-c56f-4704-91c9-bd2de3648f55-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hzgjr\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:04:59.423881 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.423850 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgt9n\" (UniqueName: \"kubernetes.io/projected/81a14fb4-c56f-4704-91c9-bd2de3648f55-kube-api-access-wgt9n\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hzgjr\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:04:59.453534 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.453401 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-msdt2\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:04:59.453727 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.453603 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-msdt2\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:04:59.454031 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.454008 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-msdt2\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:04:59.454134 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.454103 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-msdt2\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:04:59.454198 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.454155 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-msdt2\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:04:59.454254 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.454211 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-msdt2\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:04:59.454254 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.454243 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbmzs\" (UniqueName: \"kubernetes.io/projected/e3271312-1e59-4a84-8ac1-d8e57f69f85b-kube-api-access-zbmzs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-msdt2\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:04:59.454364 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.454297 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-msdt2\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:04:59.454364 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.454318 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-msdt2\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:04:59.454364 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.454340 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-msdt2\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:04:59.454526 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.454395 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-msdt2\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:04:59.454580 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.454566 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-msdt2\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:04:59.454748 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.454728 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-msdt2\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:04:59.455196 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.455159 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-msdt2\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:04:59.457361 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.457003 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-msdt2\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:04:59.457714 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.457689 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-msdt2\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:04:59.512546 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.512512 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-msdt2\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:04:59.512922 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.512896 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbmzs\" (UniqueName: \"kubernetes.io/projected/e3271312-1e59-4a84-8ac1-d8e57f69f85b-kube-api-access-zbmzs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-msdt2\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:04:59.518508 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.518475 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:04:59.576018 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.575979 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:04:59.947556 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.947521 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr"] Nov 03 21:04:59.948806 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:04:59.948769 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a14fb4_c56f_4704_91c9_bd2de3648f55.slice/crio-ca075319251623d38a89665469f86a3731c874143e3a8a6a8b8f7556cdbfc095 WatchSource:0}: Error finding container ca075319251623d38a89665469f86a3731c874143e3a8a6a8b8f7556cdbfc095: Status 404 returned error can't find the container with id ca075319251623d38a89665469f86a3731c874143e3a8a6a8b8f7556cdbfc095 Nov 03 21:04:59.951681 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.951643 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:04:59.951757 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.951734 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:04:59.951798 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.951781 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:04:59.996878 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:04:59.996849 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2"] Nov 03 21:05:00.011420 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:05:00.011389 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3271312_1e59_4a84_8ac1_d8e57f69f85b.slice/crio-b1bdc40c6ff8a1affa342546e0b61ee919282c6d3a0190f0ffe36fa5aa360311 WatchSource:0}: Error finding container b1bdc40c6ff8a1affa342546e0b61ee919282c6d3a0190f0ffe36fa5aa360311: Status 404 returned error can't find the container with id b1bdc40c6ff8a1affa342546e0b61ee919282c6d3a0190f0ffe36fa5aa360311 Nov 03 21:05:00.014316 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:00.014272 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:05:00.014400 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:00.014368 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:05:00.014450 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:00.014423 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:05:00.660009 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:00.659968 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" event={"ID":"81a14fb4-c56f-4704-91c9-bd2de3648f55","Type":"ContainerStarted","Data":"dc0acac9d72efcc6f2276daf9541b572ca901ca5699a98d6353c98c890a41a22"} Nov 03 21:05:00.670536 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:00.670496 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" event={"ID":"81a14fb4-c56f-4704-91c9-bd2de3648f55","Type":"ContainerStarted","Data":"ca075319251623d38a89665469f86a3731c874143e3a8a6a8b8f7556cdbfc095"} Nov 03 21:05:00.677788 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:00.677754 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" event={"ID":"e3271312-1e59-4a84-8ac1-d8e57f69f85b","Type":"ContainerStarted","Data":"466fda4383bb15f4883e6330f8b8a5b9a4b1f043bf2cc2210ac5744fdf2519ee"} Nov 03 21:05:00.677955 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:00.677809 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" event={"ID":"e3271312-1e59-4a84-8ac1-d8e57f69f85b","Type":"ContainerStarted","Data":"b1bdc40c6ff8a1affa342546e0b61ee919282c6d3a0190f0ffe36fa5aa360311"} Nov 03 21:05:00.793364 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:00.793296 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" podStartSLOduration=1.793273822 podStartE2EDuration="1.793273822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:00.790038508 +0000 UTC m=+995.257817235" watchObservedRunningTime="2026-11-03 21:05:00.793273822 +0000 UTC m=+995.261052546" Nov 03 21:05:00.924154 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:00.923754 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" podStartSLOduration=1.923722595 podStartE2EDuration="1.923722595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:00.920622617 +0000 UTC m=+995.388401341" watchObservedRunningTime="2026-11-03 21:05:00.923722595 +0000 UTC m=+995.391501319" Nov 03 21:05:01.520098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:01.519434 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:05:01.523215 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:01.523180 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 21:05:01.523452 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:01.523426 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" podUID="81a14fb4-c56f-4704-91c9-bd2de3648f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 21:05:01.582579 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:01.582546 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:05:01.582915 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:01.582895 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-msdt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 21:05:01.583056 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:01.583034 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" podUID="e3271312-1e59-4a84-8ac1-d8e57f69f85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 21:05:01.586222 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:01.586187 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s" podUID="6df74952-1d07-4288-8ea3-d844cb913fc8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:05:02.542100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:02.541485 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 21:05:02.542100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:02.541554 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" podUID="81a14fb4-c56f-4704-91c9-bd2de3648f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 21:05:02.576662 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:02.576623 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-msdt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 21:05:02.576814 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:02.576675 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" podUID="e3271312-1e59-4a84-8ac1-d8e57f69f85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 21:05:03.534098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:03.533466 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 21:05:03.534098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:03.533532 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" podUID="81a14fb4-c56f-4704-91c9-bd2de3648f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 21:05:03.577303 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:03.577266 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-msdt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 21:05:03.577727 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:03.577345 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" podUID="e3271312-1e59-4a84-8ac1-d8e57f69f85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 21:05:04.520099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:04.519531 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 21:05:04.520099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:04.519611 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" podUID="81a14fb4-c56f-4704-91c9-bd2de3648f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 21:05:04.577285 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:04.577253 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-msdt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 21:05:04.577493 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:04.577457 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" podUID="e3271312-1e59-4a84-8ac1-d8e57f69f85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 21:05:05.527578 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:05.527477 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 21:05:05.527814 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:05.527784 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" podUID="81a14fb4-c56f-4704-91c9-bd2de3648f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 21:05:05.576911 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:05.576878 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-msdt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 21:05:05.577064 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:05.576945 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" podUID="e3271312-1e59-4a84-8ac1-d8e57f69f85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 21:05:06.523102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:06.522356 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 21:05:06.523102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:06.522421 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" podUID="81a14fb4-c56f-4704-91c9-bd2de3648f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 21:05:06.576776 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:06.576736 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-msdt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 21:05:06.576926 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:06.576795 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" podUID="e3271312-1e59-4a84-8ac1-d8e57f69f85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 21:05:07.519561 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:07.519521 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 21:05:07.519757 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:07.519590 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" podUID="81a14fb4-c56f-4704-91c9-bd2de3648f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 21:05:07.576467 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:07.576435 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-msdt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 21:05:07.576840 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:07.576486 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" podUID="e3271312-1e59-4a84-8ac1-d8e57f69f85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 21:05:09.019120 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:09.019056 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:05:09.019635 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:09.019424 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:05:09.020489 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:09.020461 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:05:09.174994 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:09.174154 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:05:09.174994 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:09.174517 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:05:09.176130 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:09.176106 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:05:10.619083 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:05:10.619031 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:05:11.556033 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:11.555997 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s" podUID="6df74952-1d07-4288-8ea3-d844cb913fc8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:05:11.703348 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:11.703311 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6"] Nov 03 21:05:11.706684 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:11.706662 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" Nov 03 21:05:11.806149 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:11.806044 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a41ad0\"/\"distinct-w1-a41ad0-kserve-self-signed-certs\"" Nov 03 21:05:11.808330 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:11.808307 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6"] Nov 03 21:05:11.883112 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:11.883039 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ccrs\" (UniqueName: \"kubernetes.io/projected/39c75c51-eff9-4cb8-88b9-618dbb2ba188-kube-api-access-7ccrs\") pod \"distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6\" (UID: \"39c75c51-eff9-4cb8-88b9-618dbb2ba188\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" Nov 03 21:05:11.884273 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:11.884244 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39c75c51-eff9-4cb8-88b9-618dbb2ba188-dshm\") pod \"distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6\" (UID: \"39c75c51-eff9-4cb8-88b9-618dbb2ba188\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" Nov 03 21:05:11.884491 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:11.884470 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39c75c51-eff9-4cb8-88b9-618dbb2ba188-model-cache\") pod \"distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6\" (UID: \"39c75c51-eff9-4cb8-88b9-618dbb2ba188\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" Nov 03 21:05:11.886022 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:11.884731 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39c75c51-eff9-4cb8-88b9-618dbb2ba188-tls-certs\") pod \"distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6\" (UID: \"39c75c51-eff9-4cb8-88b9-618dbb2ba188\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" Nov 03 21:05:11.886273 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:11.886251 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39c75c51-eff9-4cb8-88b9-618dbb2ba188-tmp-dir\") pod \"distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6\" (UID: \"39c75c51-eff9-4cb8-88b9-618dbb2ba188\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" Nov 03 21:05:11.886447 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:11.886430 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39c75c51-eff9-4cb8-88b9-618dbb2ba188-home\") pod \"distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6\" (UID: \"39c75c51-eff9-4cb8-88b9-618dbb2ba188\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" Nov 03 21:05:11.988008 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:11.987966 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39c75c51-eff9-4cb8-88b9-618dbb2ba188-tls-certs\") pod \"distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6\" (UID: \"39c75c51-eff9-4cb8-88b9-618dbb2ba188\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" Nov 03 21:05:11.988216 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:11.988033 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39c75c51-eff9-4cb8-88b9-618dbb2ba188-tmp-dir\") pod \"distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6\" (UID: \"39c75c51-eff9-4cb8-88b9-618dbb2ba188\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" Nov 03 21:05:11.988216 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:11.988063 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39c75c51-eff9-4cb8-88b9-618dbb2ba188-home\") pod \"distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6\" (UID: \"39c75c51-eff9-4cb8-88b9-618dbb2ba188\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" Nov 03 21:05:11.988216 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:11.988104 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ccrs\" (UniqueName: \"kubernetes.io/projected/39c75c51-eff9-4cb8-88b9-618dbb2ba188-kube-api-access-7ccrs\") pod \"distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6\" (UID: \"39c75c51-eff9-4cb8-88b9-618dbb2ba188\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" Nov 03 21:05:11.988216 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:11.988192 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39c75c51-eff9-4cb8-88b9-618dbb2ba188-dshm\") pod \"distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6\" (UID: \"39c75c51-eff9-4cb8-88b9-618dbb2ba188\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" Nov 03 21:05:11.988216 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:11.988215 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39c75c51-eff9-4cb8-88b9-618dbb2ba188-model-cache\") pod \"distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6\" (UID: \"39c75c51-eff9-4cb8-88b9-618dbb2ba188\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" Nov 03 21:05:11.988528 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:11.988506 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39c75c51-eff9-4cb8-88b9-618dbb2ba188-home\") pod \"distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6\" (UID: \"39c75c51-eff9-4cb8-88b9-618dbb2ba188\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" Nov 03 21:05:11.988624 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:11.988600 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39c75c51-eff9-4cb8-88b9-618dbb2ba188-tmp-dir\") pod \"distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6\" (UID: \"39c75c51-eff9-4cb8-88b9-618dbb2ba188\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" Nov 03 21:05:11.988739 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:11.988722 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39c75c51-eff9-4cb8-88b9-618dbb2ba188-model-cache\") pod \"distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6\" (UID: \"39c75c51-eff9-4cb8-88b9-618dbb2ba188\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" Nov 03 21:05:11.990469 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:11.990435 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39c75c51-eff9-4cb8-88b9-618dbb2ba188-dshm\") pod \"distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6\" (UID: \"39c75c51-eff9-4cb8-88b9-618dbb2ba188\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" Nov 03 21:05:11.990576 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:11.990557 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39c75c51-eff9-4cb8-88b9-618dbb2ba188-tls-certs\") pod \"distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6\" (UID: \"39c75c51-eff9-4cb8-88b9-618dbb2ba188\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" Nov 03 21:05:12.047025 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:12.046987 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ccrs\" (UniqueName: \"kubernetes.io/projected/39c75c51-eff9-4cb8-88b9-618dbb2ba188-kube-api-access-7ccrs\") pod \"distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6\" (UID: \"39c75c51-eff9-4cb8-88b9-618dbb2ba188\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" Nov 03 21:05:12.318445 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:12.318399 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" Nov 03 21:05:12.685963 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:12.685926 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6"] Nov 03 21:05:12.687270 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:05:12.687234 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39c75c51_eff9_4cb8_88b9_618dbb2ba188.slice/crio-63f7aa919b27c56c9f2949e93b478877903ef25da169174508fed4ea9fa235a0 WatchSource:0}: Error finding container 63f7aa919b27c56c9f2949e93b478877903ef25da169174508fed4ea9fa235a0: Status 404 returned error can't find the container with id 63f7aa919b27c56c9f2949e93b478877903ef25da169174508fed4ea9fa235a0 Nov 03 21:05:12.822149 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:12.822111 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" event={"ID":"39c75c51-eff9-4cb8-88b9-618dbb2ba188","Type":"ContainerStarted","Data":"9726c702cffdc192c8b51ce7a0125d5609839491dae2879630182a0a4d175614"} Nov 03 21:05:12.822559 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:12.822160 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" event={"ID":"39c75c51-eff9-4cb8-88b9-618dbb2ba188","Type":"ContainerStarted","Data":"63f7aa919b27c56c9f2949e93b478877903ef25da169174508fed4ea9fa235a0"} Nov 03 21:05:12.934497 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:12.934381 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" podStartSLOduration=1.934363659 podStartE2EDuration="1.934363659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:12.933786328 +0000 UTC m=+1007.401565054" watchObservedRunningTime="2026-11-03 21:05:12.934363659 +0000 UTC m=+1007.402142383" Nov 03 21:05:13.319216 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:13.319174 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" Nov 03 21:05:21.555784 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:21.555738 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s" podUID="6df74952-1d07-4288-8ea3-d844cb913fc8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:05:22.318939 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:22.318903 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" Nov 03 21:05:22.334838 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:22.334811 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" Nov 03 21:05:22.886504 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:22.886474 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" Nov 03 21:05:31.555504 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:31.555454 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s" podUID="6df74952-1d07-4288-8ea3-d844cb913fc8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:05:40.619471 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:05:40.619435 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:05:41.555343 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:41.555301 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s" podUID="6df74952-1d07-4288-8ea3-d844cb913fc8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:05:51.555560 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:51.555517 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s" podUID="6df74952-1d07-4288-8ea3-d844cb913fc8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:05:53.780534 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:53.780503 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s"] Nov 03 21:05:53.781818 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:53.781789 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s" podUID="6df74952-1d07-4288-8ea3-d844cb913fc8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:05:53.812488 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:53.812447 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4"] Nov 03 21:05:53.816769 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:53.816748 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4" Nov 03 21:05:53.866740 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:53.866708 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b3ede8887360\"" Nov 03 21:05:53.886575 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:53.886542 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4"] Nov 03 21:05:53.909628 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:53.909585 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b53e1266-9fb0-4047-a962-6cb1e2b4d981-routing-overlay-volume\") pod \"payload-processing-external-model-7cd67c9464-7wrt4\" (UID: \"b53e1266-9fb0-4047-a962-6cb1e2b4d981\") " pod="models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4" Nov 03 21:05:53.909801 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:53.909743 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5z5h\" (UniqueName: \"kubernetes.io/projected/b53e1266-9fb0-4047-a962-6cb1e2b4d981-kube-api-access-l5z5h\") pod \"payload-processing-external-model-7cd67c9464-7wrt4\" (UID: \"b53e1266-9fb0-4047-a962-6cb1e2b4d981\") " pod="models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4" Nov 03 21:05:53.909801 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:53.909773 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b53e1266-9fb0-4047-a962-6cb1e2b4d981-plugins-config-volume\") pod \"payload-processing-external-model-7cd67c9464-7wrt4\" (UID: \"b53e1266-9fb0-4047-a962-6cb1e2b4d981\") " pod="models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4" Nov 03 21:05:53.909899 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:53.909879 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b53e1266-9fb0-4047-a962-6cb1e2b4d981-provider-credentials\") pod \"payload-processing-external-model-7cd67c9464-7wrt4\" (UID: \"b53e1266-9fb0-4047-a962-6cb1e2b4d981\") " pod="models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4" Nov 03 21:05:54.012905 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:54.012851 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b53e1266-9fb0-4047-a962-6cb1e2b4d981-routing-overlay-volume\") pod \"payload-processing-external-model-7cd67c9464-7wrt4\" (UID: \"b53e1266-9fb0-4047-a962-6cb1e2b4d981\") " pod="models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4" Nov 03 21:05:54.014262 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:54.014104 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b53e1266-9fb0-4047-a962-6cb1e2b4d981-routing-overlay-volume\") pod \"payload-processing-external-model-7cd67c9464-7wrt4\" (UID: \"b53e1266-9fb0-4047-a962-6cb1e2b4d981\") " pod="models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4" Nov 03 21:05:54.016265 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:54.014283 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5z5h\" (UniqueName: \"kubernetes.io/projected/b53e1266-9fb0-4047-a962-6cb1e2b4d981-kube-api-access-l5z5h\") pod \"payload-processing-external-model-7cd67c9464-7wrt4\" (UID: \"b53e1266-9fb0-4047-a962-6cb1e2b4d981\") " pod="models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4" Nov 03 21:05:54.016534 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:54.016513 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b53e1266-9fb0-4047-a962-6cb1e2b4d981-plugins-config-volume\") pod \"payload-processing-external-model-7cd67c9464-7wrt4\" (UID: \"b53e1266-9fb0-4047-a962-6cb1e2b4d981\") " pod="models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4" Nov 03 21:05:54.017160 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:54.017131 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b53e1266-9fb0-4047-a962-6cb1e2b4d981-provider-credentials\") pod \"payload-processing-external-model-7cd67c9464-7wrt4\" (UID: \"b53e1266-9fb0-4047-a962-6cb1e2b4d981\") " pod="models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4" Nov 03 21:05:54.018174 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:54.018146 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b53e1266-9fb0-4047-a962-6cb1e2b4d981-plugins-config-volume\") pod \"payload-processing-external-model-7cd67c9464-7wrt4\" (UID: \"b53e1266-9fb0-4047-a962-6cb1e2b4d981\") " pod="models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4" Nov 03 21:05:54.020385 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:54.020361 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b53e1266-9fb0-4047-a962-6cb1e2b4d981-provider-credentials\") pod \"payload-processing-external-model-7cd67c9464-7wrt4\" (UID: \"b53e1266-9fb0-4047-a962-6cb1e2b4d981\") " pod="models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4" Nov 03 21:05:54.081344 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:54.081245 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5z5h\" (UniqueName: \"kubernetes.io/projected/b53e1266-9fb0-4047-a962-6cb1e2b4d981-kube-api-access-l5z5h\") pod \"payload-processing-external-model-7cd67c9464-7wrt4\" (UID: \"b53e1266-9fb0-4047-a962-6cb1e2b4d981\") " pod="models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4" Nov 03 21:05:54.132610 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:54.132573 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4" Nov 03 21:05:54.507802 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:54.507778 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4"] Nov 03 21:05:54.509601 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:05:54.509570 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb53e1266_9fb0_4047_a962_6cb1e2b4d981.slice/crio-4bcda6b3c2f5e2f7dfa935f58af27016773f3f6548e1f25160ae1eeb53a64d13 WatchSource:0}: Error finding container 4bcda6b3c2f5e2f7dfa935f58af27016773f3f6548e1f25160ae1eeb53a64d13: Status 404 returned error can't find the container with id 4bcda6b3c2f5e2f7dfa935f58af27016773f3f6548e1f25160ae1eeb53a64d13 Nov 03 21:05:55.093850 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:55.093809 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4" event={"ID":"b53e1266-9fb0-4047-a962-6cb1e2b4d981","Type":"ContainerStarted","Data":"568a47fe3b014e5eb07d17e57101cc1e280f53a518ce16b1d0555d1c484c8578"} Nov 03 21:05:55.094321 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:55.093873 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4" event={"ID":"b53e1266-9fb0-4047-a962-6cb1e2b4d981","Type":"ContainerStarted","Data":"4bcda6b3c2f5e2f7dfa935f58af27016773f3f6548e1f25160ae1eeb53a64d13"} Nov 03 21:05:55.094321 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:55.093981 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4" Nov 03 21:05:55.204752 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:55.204700 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4" podStartSLOduration=2.204686252 podStartE2EDuration="2.204686252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:55.204522566 +0000 UTC m=+1049.672301289" watchObservedRunningTime="2026-11-03 21:05:55.204686252 +0000 UTC m=+1049.672464975" Nov 03 21:05:58.781111 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:58.781040 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s" podUID="6df74952-1d07-4288-8ea3-d844cb913fc8" containerName="payload-processing" containerID="cri-o://58a5027d0d849478376e1ebed780da452366847434eb66d3576724df71696aff" gracePeriod=25 Nov 03 21:05:59.061705 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:59.061679 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s" Nov 03 21:05:59.119292 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:59.119260 5615 generic.go:356] "Generic (PLEG): container finished" podID="6df74952-1d07-4288-8ea3-d844cb913fc8" containerID="58a5027d0d849478376e1ebed780da452366847434eb66d3576724df71696aff" exitCode=0 Nov 03 21:05:59.119453 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:59.119304 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s" Nov 03 21:05:59.119453 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:59.119340 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s" event={"ID":"6df74952-1d07-4288-8ea3-d844cb913fc8","Type":"ContainerDied","Data":"58a5027d0d849478376e1ebed780da452366847434eb66d3576724df71696aff"} Nov 03 21:05:59.119453 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:59.119394 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s" event={"ID":"6df74952-1d07-4288-8ea3-d844cb913fc8","Type":"ContainerDied","Data":"a2e038eb5f44d9c52d3e5d1823c0dec3c6b0a8bc025c1a9194c26492fd8cd5aa"} Nov 03 21:05:59.119453 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:59.119424 5615 scope.go:122] "RemoveContainer" containerID="58a5027d0d849478376e1ebed780da452366847434eb66d3576724df71696aff" Nov 03 21:05:59.128984 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:59.128962 5615 scope.go:122] "RemoveContainer" containerID="58a5027d0d849478376e1ebed780da452366847434eb66d3576724df71696aff" Nov 03 21:05:59.129315 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:05:59.129294 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58a5027d0d849478376e1ebed780da452366847434eb66d3576724df71696aff\": container with ID starting with 58a5027d0d849478376e1ebed780da452366847434eb66d3576724df71696aff not found: ID does not exist" containerID="58a5027d0d849478376e1ebed780da452366847434eb66d3576724df71696aff" Nov 03 21:05:59.129395 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:59.129330 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58a5027d0d849478376e1ebed780da452366847434eb66d3576724df71696aff"} err="failed to get container status \"58a5027d0d849478376e1ebed780da452366847434eb66d3576724df71696aff\": rpc error: code = NotFound desc = could not find container \"58a5027d0d849478376e1ebed780da452366847434eb66d3576724df71696aff\": container with ID starting with 58a5027d0d849478376e1ebed780da452366847434eb66d3576724df71696aff not found: ID does not exist" Nov 03 21:05:59.185218 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:59.185177 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df74952-1d07-4288-8ea3-d844cb913fc8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6df74952-1d07-4288-8ea3-d844cb913fc8-provider-credentials\") pod \"6df74952-1d07-4288-8ea3-d844cb913fc8\" (UID: \"6df74952-1d07-4288-8ea3-d844cb913fc8\") " Nov 03 21:05:59.185401 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:59.185274 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6df74952-1d07-4288-8ea3-d844cb913fc8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df74952-1d07-4288-8ea3-d844cb913fc8-plugins-config-volume\") pod \"6df74952-1d07-4288-8ea3-d844cb913fc8\" (UID: \"6df74952-1d07-4288-8ea3-d844cb913fc8\") " Nov 03 21:05:59.185401 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:59.185383 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df74952-1d07-4288-8ea3-d844cb913fc8-kube-api-access-v92lv\" (UniqueName: \"kubernetes.io/projected/6df74952-1d07-4288-8ea3-d844cb913fc8-kube-api-access-v92lv\") pod \"6df74952-1d07-4288-8ea3-d844cb913fc8\" (UID: \"6df74952-1d07-4288-8ea3-d844cb913fc8\") " Nov 03 21:05:59.185482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:59.185412 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6df74952-1d07-4288-8ea3-d844cb913fc8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6df74952-1d07-4288-8ea3-d844cb913fc8-routing-overlay-volume\") pod \"6df74952-1d07-4288-8ea3-d844cb913fc8\" (UID: \"6df74952-1d07-4288-8ea3-d844cb913fc8\") " Nov 03 21:05:59.185763 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:59.185738 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df74952-1d07-4288-8ea3-d844cb913fc8-plugins-config-volume" pod "6df74952-1d07-4288-8ea3-d844cb913fc8" (UID: "6df74952-1d07-4288-8ea3-d844cb913fc8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:59.185860 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:59.185814 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df74952-1d07-4288-8ea3-d844cb913fc8-routing-overlay-volume" pod "6df74952-1d07-4288-8ea3-d844cb913fc8" (UID: "6df74952-1d07-4288-8ea3-d844cb913fc8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:59.185977 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:59.185960 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df74952-1d07-4288-8ea3-d844cb913fc8-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:59.186035 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:59.185979 5615 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6df74952-1d07-4288-8ea3-d844cb913fc8-routing-overlay-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:59.187665 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:59.187642 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df74952-1d07-4288-8ea3-d844cb913fc8-provider-credentials" pod "6df74952-1d07-4288-8ea3-d844cb913fc8" (UID: "6df74952-1d07-4288-8ea3-d844cb913fc8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:59.187784 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:59.187730 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df74952-1d07-4288-8ea3-d844cb913fc8-kube-api-access-v92lv" pod "6df74952-1d07-4288-8ea3-d844cb913fc8" (UID: "6df74952-1d07-4288-8ea3-d844cb913fc8"). InnerVolumeSpecName "kube-api-access-v92lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:59.286836 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:59.286751 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v92lv\" (UniqueName: \"kubernetes.io/projected/6df74952-1d07-4288-8ea3-d844cb913fc8-kube-api-access-v92lv\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:59.286836 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:59.286783 5615 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6df74952-1d07-4288-8ea3-d844cb913fc8-provider-credentials\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:59.534634 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:59.534599 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s"] Nov 03 21:05:59.541812 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:59.541737 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-56cc46bb7d-58w9s"] Nov 03 21:05:59.655250 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:05:59.655216 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6df74952-1d07-4288-8ea3-d844cb913fc8" path="/var/lib/kubelet/pods/6df74952-1d07-4288-8ea3-d844cb913fc8/volumes" Nov 03 21:06:04.527907 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:04.527867 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc"] Nov 03 21:06:04.530273 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:04.530248 5615 cpu_manager.go:414] "containerMap: removing container" podUID="6df74952-1d07-4288-8ea3-d844cb913fc8" containerName="payload-processing" Nov 03 21:06:04.530458 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:04.530442 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df74952-1d07-4288-8ea3-d844cb913fc8" containerName="payload-processing" Nov 03 21:06:04.530748 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:04.530731 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df74952-1d07-4288-8ea3-d844cb913fc8" containerName="payload-processing" Nov 03 21:06:04.547171 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:04.540682 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:04.668135 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:04.668059 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc"] Nov 03 21:06:04.723466 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:04.723413 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:04.723818 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:04.723795 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:04.885790 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:04.885688 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjlm6\" (UniqueName: \"kubernetes.io/projected/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-kube-api-access-pjlm6\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:04.886238 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:04.886211 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:04.886496 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:04.886472 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:04.886676 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:04.886657 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:04.886829 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:04.886811 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:04.886967 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:04.886950 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:04.887113 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:04.887094 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:05.067570 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:05.067528 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:05.068345 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:05.068316 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:05.068999 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:05.068970 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjlm6\" (UniqueName: \"kubernetes.io/projected/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-kube-api-access-pjlm6\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:05.069693 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:05.069672 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:05.070106 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:05.070069 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:05.070268 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:05.070254 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:05.070446 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:05.070377 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:05.070580 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:05.070563 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:05.070682 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:05.070670 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:05.072843 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:05.068251 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:05.072985 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:05.071290 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:05.076542 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:05.071506 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:05.076690 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:05.072104 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:05.076801 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:05.072327 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:05.140034 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:05.139935 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:05.192229 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:05.192157 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjlm6\" (UniqueName: \"kubernetes.io/projected/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-kube-api-access-pjlm6\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:05.193864 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:05.193791 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:05.195516 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:05.194461 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:05.478514 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:05.478400 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:06.292109 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:06.291231 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4" podUID="b53e1266-9fb0-4047-a962-6cb1e2b4d981" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:06:06.427223 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:06.427108 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc"] Nov 03 21:06:06.602285 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:06.602234 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:06:06.609316 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:06.609274 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:06:06.609554 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:06.609522 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:06:07.347604 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:07.347562 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" event={"ID":"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f","Type":"ContainerStarted","Data":"28d10fc2e4fde65a1406a814a096bc131cfa2fd91d26109f5756b86de88e8602"} Nov 03 21:06:07.348113 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:07.347632 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" event={"ID":"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f","Type":"ContainerStarted","Data":"efa46e3eb9a2fead4f23cc031d29bdf59e91947d9b80e997862ff6b5a27475f0"} Nov 03 21:06:07.480115 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:07.480063 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:07.481669 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:07.481642 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 21:06:07.481795 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:07.481696 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" podUID="6a9c6e9e-a20e-40e4-a4a1-207a92d7375f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 21:06:07.562515 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:07.562452 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" podStartSLOduration=3.562437376 podStartE2EDuration="3.562437376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:07.513573389 +0000 UTC m=+1061.981352112" watchObservedRunningTime="2026-11-03 21:06:07.562437376 +0000 UTC m=+1062.030216160" Nov 03 21:06:08.480257 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:08.480219 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 21:06:08.480641 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:08.480286 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" podUID="6a9c6e9e-a20e-40e4-a4a1-207a92d7375f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 21:06:09.480288 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:09.480245 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 21:06:09.480695 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:09.480302 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" podUID="6a9c6e9e-a20e-40e4-a4a1-207a92d7375f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 21:06:09.533143 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:09.533102 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4"] Nov 03 21:06:09.534529 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:09.534501 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4" podUID="b53e1266-9fb0-4047-a962-6cb1e2b4d981" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:06:10.479527 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:10.479490 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 21:06:10.479693 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:10.479548 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" podUID="6a9c6e9e-a20e-40e4-a4a1-207a92d7375f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 21:06:10.642207 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:06:10.642170 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:06:11.480144 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:11.480102 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 21:06:11.480336 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:11.480160 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" podUID="6a9c6e9e-a20e-40e4-a4a1-207a92d7375f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 21:06:12.479912 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:12.479877 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 21:06:12.480356 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:12.479943 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" podUID="6a9c6e9e-a20e-40e4-a4a1-207a92d7375f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 21:06:13.761520 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:13.761385 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:13.762919 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:13.762858 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:13.764367 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:13.764345 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:06:14.533725 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:14.533677 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4" podUID="b53e1266-9fb0-4047-a962-6cb1e2b4d981" containerName="payload-processing" containerID="cri-o://568a47fe3b014e5eb07d17e57101cc1e280f53a518ce16b1d0555d1c484c8578" gracePeriod=25 Nov 03 21:06:14.799132 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:14.799038 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4" Nov 03 21:06:14.881478 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:14.881441 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b53e1266-9fb0-4047-a962-6cb1e2b4d981-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b53e1266-9fb0-4047-a962-6cb1e2b4d981-routing-overlay-volume\") pod \"b53e1266-9fb0-4047-a962-6cb1e2b4d981\" (UID: \"b53e1266-9fb0-4047-a962-6cb1e2b4d981\") " Nov 03 21:06:14.881663 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:14.881508 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b53e1266-9fb0-4047-a962-6cb1e2b4d981-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b53e1266-9fb0-4047-a962-6cb1e2b4d981-provider-credentials\") pod \"b53e1266-9fb0-4047-a962-6cb1e2b4d981\" (UID: \"b53e1266-9fb0-4047-a962-6cb1e2b4d981\") " Nov 03 21:06:14.881663 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:14.881606 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b53e1266-9fb0-4047-a962-6cb1e2b4d981-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b53e1266-9fb0-4047-a962-6cb1e2b4d981-plugins-config-volume\") pod \"b53e1266-9fb0-4047-a962-6cb1e2b4d981\" (UID: \"b53e1266-9fb0-4047-a962-6cb1e2b4d981\") " Nov 03 21:06:14.881796 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:14.881713 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b53e1266-9fb0-4047-a962-6cb1e2b4d981-kube-api-access-l5z5h\" (UniqueName: \"kubernetes.io/projected/b53e1266-9fb0-4047-a962-6cb1e2b4d981-kube-api-access-l5z5h\") pod \"b53e1266-9fb0-4047-a962-6cb1e2b4d981\" (UID: \"b53e1266-9fb0-4047-a962-6cb1e2b4d981\") " Nov 03 21:06:14.881962 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:14.881932 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b53e1266-9fb0-4047-a962-6cb1e2b4d981-routing-overlay-volume" pod "b53e1266-9fb0-4047-a962-6cb1e2b4d981" (UID: "b53e1266-9fb0-4047-a962-6cb1e2b4d981"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:06:14.882022 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:14.881969 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b53e1266-9fb0-4047-a962-6cb1e2b4d981-plugins-config-volume" pod "b53e1266-9fb0-4047-a962-6cb1e2b4d981" (UID: "b53e1266-9fb0-4047-a962-6cb1e2b4d981"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:06:14.882127 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:14.882107 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b53e1266-9fb0-4047-a962-6cb1e2b4d981-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:14.882127 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:14.882126 5615 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b53e1266-9fb0-4047-a962-6cb1e2b4d981-routing-overlay-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:14.883812 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:14.883785 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b53e1266-9fb0-4047-a962-6cb1e2b4d981-kube-api-access-l5z5h" pod "b53e1266-9fb0-4047-a962-6cb1e2b4d981" (UID: "b53e1266-9fb0-4047-a962-6cb1e2b4d981"). InnerVolumeSpecName "kube-api-access-l5z5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:14.883897 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:14.883822 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b53e1266-9fb0-4047-a962-6cb1e2b4d981-provider-credentials" pod "b53e1266-9fb0-4047-a962-6cb1e2b4d981" (UID: "b53e1266-9fb0-4047-a962-6cb1e2b4d981"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:14.985965 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:14.985928 5615 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b53e1266-9fb0-4047-a962-6cb1e2b4d981-provider-credentials\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:14.986254 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:14.986234 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5z5h\" (UniqueName: \"kubernetes.io/projected/b53e1266-9fb0-4047-a962-6cb1e2b4d981-kube-api-access-l5z5h\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:15.408950 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:15.408905 5615 generic.go:356] "Generic (PLEG): container finished" podID="b53e1266-9fb0-4047-a962-6cb1e2b4d981" containerID="568a47fe3b014e5eb07d17e57101cc1e280f53a518ce16b1d0555d1c484c8578" exitCode=0 Nov 03 21:06:15.409315 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:15.409290 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4" Nov 03 21:06:15.409747 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:15.409716 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4" event={"ID":"b53e1266-9fb0-4047-a962-6cb1e2b4d981","Type":"ContainerDied","Data":"568a47fe3b014e5eb07d17e57101cc1e280f53a518ce16b1d0555d1c484c8578"} Nov 03 21:06:15.409854 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:15.409776 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4" event={"ID":"b53e1266-9fb0-4047-a962-6cb1e2b4d981","Type":"ContainerDied","Data":"4bcda6b3c2f5e2f7dfa935f58af27016773f3f6548e1f25160ae1eeb53a64d13"} Nov 03 21:06:15.409854 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:15.409808 5615 scope.go:122] "RemoveContainer" containerID="568a47fe3b014e5eb07d17e57101cc1e280f53a518ce16b1d0555d1c484c8578" Nov 03 21:06:15.424671 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:15.424645 5615 scope.go:122] "RemoveContainer" containerID="568a47fe3b014e5eb07d17e57101cc1e280f53a518ce16b1d0555d1c484c8578" Nov 03 21:06:15.425162 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:06:15.425128 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"568a47fe3b014e5eb07d17e57101cc1e280f53a518ce16b1d0555d1c484c8578\": container with ID starting with 568a47fe3b014e5eb07d17e57101cc1e280f53a518ce16b1d0555d1c484c8578 not found: ID does not exist" containerID="568a47fe3b014e5eb07d17e57101cc1e280f53a518ce16b1d0555d1c484c8578" Nov 03 21:06:15.425279 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:15.425174 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"568a47fe3b014e5eb07d17e57101cc1e280f53a518ce16b1d0555d1c484c8578"} err="failed to get container status \"568a47fe3b014e5eb07d17e57101cc1e280f53a518ce16b1d0555d1c484c8578\": rpc error: code = NotFound desc = could not find container \"568a47fe3b014e5eb07d17e57101cc1e280f53a518ce16b1d0555d1c484c8578\": container with ID starting with 568a47fe3b014e5eb07d17e57101cc1e280f53a518ce16b1d0555d1c484c8578 not found: ID does not exist" Nov 03 21:06:15.569437 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:15.569395 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4"] Nov 03 21:06:15.616242 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:15.616198 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cd67c9464-7wrt4"] Nov 03 21:06:15.656667 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:15.656618 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b53e1266-9fb0-4047-a962-6cb1e2b4d981" path="/var/lib/kubelet/pods/b53e1266-9fb0-4047-a962-6cb1e2b4d981/volumes" Nov 03 21:06:25.840680 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:06:25.840646 5615 scope.go:122] "RemoveContainer" containerID="6e1ca4e19799be5f61b4f5b34235bc0f16b44b723b31723ece34ba3481ef8f88" Nov 03 21:06:40.651365 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:06:40.651253 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:07:05.810840 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:05.810797 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7"] Nov 03 21:07:05.811600 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:05.811578 5615 cpu_manager.go:414] "containerMap: removing container" podUID="b53e1266-9fb0-4047-a962-6cb1e2b4d981" containerName="payload-processing" Nov 03 21:07:05.811703 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:05.811602 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b53e1266-9fb0-4047-a962-6cb1e2b4d981" containerName="payload-processing" Nov 03 21:07:05.811761 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:05.811743 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="b53e1266-9fb0-4047-a962-6cb1e2b4d981" containerName="payload-processing" Nov 03 21:07:05.815187 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:05.815163 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" Nov 03 21:07:05.867201 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:05.867151 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-791c32\"/\"kube-root-ca.crt\"" Nov 03 21:07:05.914623 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:05.914591 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-791c32\"/\"openshift-service-ca.crt\"" Nov 03 21:07:05.914813 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:05.914638 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-791c32\"/\"distinct-2-w0-791c32-kserve-self-signed-certs\"" Nov 03 21:07:05.914813 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:05.914645 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-791c32\"/\"default-dockercfg-t5gcp\"" Nov 03 21:07:05.916850 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:05.916830 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7"] Nov 03 21:07:05.997102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:05.969566 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/232c2524-2557-47c5-9d80-f6235972ca83-home\") pod \"distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7\" (UID: \"232c2524-2557-47c5-9d80-f6235972ca83\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" Nov 03 21:07:05.998245 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:05.998215 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/232c2524-2557-47c5-9d80-f6235972ca83-tmp-dir\") pod \"distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7\" (UID: \"232c2524-2557-47c5-9d80-f6235972ca83\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" Nov 03 21:07:05.998567 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:05.998547 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/232c2524-2557-47c5-9d80-f6235972ca83-dshm\") pod \"distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7\" (UID: \"232c2524-2557-47c5-9d80-f6235972ca83\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" Nov 03 21:07:05.998756 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:05.998739 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/232c2524-2557-47c5-9d80-f6235972ca83-model-cache\") pod \"distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7\" (UID: \"232c2524-2557-47c5-9d80-f6235972ca83\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" Nov 03 21:07:05.998870 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:05.998855 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccpm5\" (UniqueName: \"kubernetes.io/projected/232c2524-2557-47c5-9d80-f6235972ca83-kube-api-access-ccpm5\") pod \"distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7\" (UID: \"232c2524-2557-47c5-9d80-f6235972ca83\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" Nov 03 21:07:05.999018 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:05.999002 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/232c2524-2557-47c5-9d80-f6235972ca83-tls-certs\") pod \"distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7\" (UID: \"232c2524-2557-47c5-9d80-f6235972ca83\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" Nov 03 21:07:06.100120 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:06.100011 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/232c2524-2557-47c5-9d80-f6235972ca83-model-cache\") pod \"distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7\" (UID: \"232c2524-2557-47c5-9d80-f6235972ca83\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" Nov 03 21:07:06.100120 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:06.100059 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccpm5\" (UniqueName: \"kubernetes.io/projected/232c2524-2557-47c5-9d80-f6235972ca83-kube-api-access-ccpm5\") pod \"distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7\" (UID: \"232c2524-2557-47c5-9d80-f6235972ca83\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" Nov 03 21:07:06.100374 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:06.100129 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/232c2524-2557-47c5-9d80-f6235972ca83-tls-certs\") pod \"distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7\" (UID: \"232c2524-2557-47c5-9d80-f6235972ca83\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" Nov 03 21:07:06.100374 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:06.100192 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/232c2524-2557-47c5-9d80-f6235972ca83-home\") pod \"distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7\" (UID: \"232c2524-2557-47c5-9d80-f6235972ca83\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" Nov 03 21:07:06.100374 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:06.100220 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/232c2524-2557-47c5-9d80-f6235972ca83-tmp-dir\") pod \"distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7\" (UID: \"232c2524-2557-47c5-9d80-f6235972ca83\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" Nov 03 21:07:06.100374 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:06.100300 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/232c2524-2557-47c5-9d80-f6235972ca83-dshm\") pod \"distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7\" (UID: \"232c2524-2557-47c5-9d80-f6235972ca83\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" Nov 03 21:07:06.100614 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:06.100588 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/232c2524-2557-47c5-9d80-f6235972ca83-model-cache\") pod \"distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7\" (UID: \"232c2524-2557-47c5-9d80-f6235972ca83\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" Nov 03 21:07:06.100673 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:06.100639 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/232c2524-2557-47c5-9d80-f6235972ca83-home\") pod \"distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7\" (UID: \"232c2524-2557-47c5-9d80-f6235972ca83\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" Nov 03 21:07:06.100769 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:06.100739 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/232c2524-2557-47c5-9d80-f6235972ca83-tmp-dir\") pod \"distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7\" (UID: \"232c2524-2557-47c5-9d80-f6235972ca83\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" Nov 03 21:07:06.102537 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:06.102518 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/232c2524-2557-47c5-9d80-f6235972ca83-dshm\") pod \"distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7\" (UID: \"232c2524-2557-47c5-9d80-f6235972ca83\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" Nov 03 21:07:06.102848 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:06.102830 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/232c2524-2557-47c5-9d80-f6235972ca83-tls-certs\") pod \"distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7\" (UID: \"232c2524-2557-47c5-9d80-f6235972ca83\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" Nov 03 21:07:06.158706 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:06.158674 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccpm5\" (UniqueName: \"kubernetes.io/projected/232c2524-2557-47c5-9d80-f6235972ca83-kube-api-access-ccpm5\") pod \"distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7\" (UID: \"232c2524-2557-47c5-9d80-f6235972ca83\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" Nov 03 21:07:06.428157 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:06.428055 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" Nov 03 21:07:06.799956 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:06.799923 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7"] Nov 03 21:07:06.802207 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:07:06.802177 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod232c2524_2557_47c5_9d80_f6235972ca83.slice/crio-ac222e3258aaea13cc488e13570f9e444aaf48fb7e2bfd531300628dee66f426 WatchSource:0}: Error finding container ac222e3258aaea13cc488e13570f9e444aaf48fb7e2bfd531300628dee66f426: Status 404 returned error can't find the container with id ac222e3258aaea13cc488e13570f9e444aaf48fb7e2bfd531300628dee66f426 Nov 03 21:07:06.888294 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:06.888257 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" event={"ID":"232c2524-2557-47c5-9d80-f6235972ca83","Type":"ContainerStarted","Data":"891b1f62cb98d0af94477919b238dd7aac6037692825823816da649f63ddc201"} Nov 03 21:07:06.888294 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:06.888301 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" event={"ID":"232c2524-2557-47c5-9d80-f6235972ca83","Type":"ContainerStarted","Data":"ac222e3258aaea13cc488e13570f9e444aaf48fb7e2bfd531300628dee66f426"} Nov 03 21:07:07.000090 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:07.000022 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" podStartSLOduration=2.000007204 podStartE2EDuration="2.000007204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:06.999096557 +0000 UTC m=+1121.466875282" watchObservedRunningTime="2026-11-03 21:07:07.000007204 +0000 UTC m=+1121.467785948" Nov 03 21:07:07.428872 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:07.428836 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" Nov 03 21:07:10.647266 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:07:10.647173 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:07:16.428235 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:16.428200 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" Nov 03 21:07:16.443033 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:16.443003 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" Nov 03 21:07:16.996446 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:16.996416 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" Nov 03 21:07:32.918157 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:32.918120 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq"] Nov 03 21:07:32.924053 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:32.924032 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:32.992334 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:32.992302 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:32.992530 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:32.992341 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:32.992530 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:32.992427 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:32.992530 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:32.992500 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc5w8\" (UniqueName: \"kubernetes.io/projected/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-kube-api-access-vc5w8\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:32.992714 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:32.992538 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:32.992714 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:32.992608 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:32.992816 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:32.992758 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:32.992816 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:32.992789 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:32.992925 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:32.992880 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:33.020320 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:33.020284 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq"] Nov 03 21:07:33.093946 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:33.093911 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:33.094134 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:33.094016 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:33.094134 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:33.094036 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:33.094134 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:33.094065 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:33.094307 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:33.094231 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:33.094307 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:33.094283 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:33.094408 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:33.094370 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:33.094464 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:33.094417 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc5w8\" (UniqueName: \"kubernetes.io/projected/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-kube-api-access-vc5w8\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:33.094464 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:33.094444 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:33.094464 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:33.094457 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:33.094613 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:33.094516 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:33.094613 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:33.094552 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:33.094731 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:33.094702 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:33.094900 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:33.094874 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:33.096323 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:33.096302 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:33.096739 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:33.096722 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:33.153303 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:33.153269 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:33.153448 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:33.153385 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc5w8\" (UniqueName: \"kubernetes.io/projected/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-kube-api-access-vc5w8\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:33.237578 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:33.237534 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:33.662960 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:33.662935 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq"] Nov 03 21:07:33.664809 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:07:33.664779 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1a6e0e_b6e7_4e5c_9ae2_738cdc0bd2c8.slice/crio-494efbd4a4575d34ad923a18c21e8a9fda42ce3bff1fc9127c8436ad7877ec11 WatchSource:0}: Error finding container 494efbd4a4575d34ad923a18c21e8a9fda42ce3bff1fc9127c8436ad7877ec11: Status 404 returned error can't find the container with id 494efbd4a4575d34ad923a18c21e8a9fda42ce3bff1fc9127c8436ad7877ec11 Nov 03 21:07:33.667271 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:33.667235 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:07:33.667392 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:33.667320 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:07:33.667392 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:33.667349 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:07:34.091925 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:34.091883 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" event={"ID":"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8","Type":"ContainerStarted","Data":"d8a08f375ad6ea2cb8202a455a89bd1628d641df2cd3a0b43f724b191349b71b"} Nov 03 21:07:34.092369 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:34.091971 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" event={"ID":"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8","Type":"ContainerStarted","Data":"494efbd4a4575d34ad923a18c21e8a9fda42ce3bff1fc9127c8436ad7877ec11"} Nov 03 21:07:34.238480 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:34.238439 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:34.240186 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:34.240159 5615 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 21:07:34.240304 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:34.240209 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" podUID="7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 21:07:35.238020 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.237985 5615 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 21:07:35.238446 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.238041 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" podUID="7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 21:07:35.252975 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.252921 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" podStartSLOduration=3.252906619 podStartE2EDuration="3.252906619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:34.222420439 +0000 UTC m=+1148.690199163" watchObservedRunningTime="2026-11-03 21:07:35.252906619 +0000 UTC m=+1149.720685341" Nov 03 21:07:35.253886 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.253867 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv"] Nov 03 21:07:35.257891 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.257868 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:35.341976 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.341841 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e81a25cc-db37-460c-8adb-9cfd35690280-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:35.342303 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.342273 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e81a25cc-db37-460c-8adb-9cfd35690280-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:35.342543 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.342522 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drb25\" (UniqueName: \"kubernetes.io/projected/e81a25cc-db37-460c-8adb-9cfd35690280-kube-api-access-drb25\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:35.342716 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.342701 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:35.342915 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.342896 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:35.343106 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.343063 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:35.343286 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.343271 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e81a25cc-db37-460c-8adb-9cfd35690280-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:35.343438 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.343422 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:35.343634 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.343616 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:35.416542 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.416498 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv"] Nov 03 21:07:35.445455 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.445389 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drb25\" (UniqueName: \"kubernetes.io/projected/e81a25cc-db37-460c-8adb-9cfd35690280-kube-api-access-drb25\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:35.445693 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.445674 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:35.445818 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.445808 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:35.445892 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.445883 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:35.445993 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.445969 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:35.446033 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.445980 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e81a25cc-db37-460c-8adb-9cfd35690280-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:35.446066 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.446031 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:35.446133 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.446100 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:35.446206 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.446190 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e81a25cc-db37-460c-8adb-9cfd35690280-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:35.446272 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.446257 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e81a25cc-db37-460c-8adb-9cfd35690280-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:35.446382 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.446359 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:35.446726 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.446706 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:35.446867 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.446839 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:35.447190 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.447170 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e81a25cc-db37-460c-8adb-9cfd35690280-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:35.448786 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.448770 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:35.448879 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.448854 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e81a25cc-db37-460c-8adb-9cfd35690280-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:35.524250 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.524159 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drb25\" (UniqueName: \"kubernetes.io/projected/e81a25cc-db37-460c-8adb-9cfd35690280-kube-api-access-drb25\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:35.524626 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.524606 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e81a25cc-db37-460c-8adb-9cfd35690280-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:35.572811 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:35.572770 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:36.027914 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:36.027884 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv"] Nov 03 21:07:36.029745 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:07:36.029711 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode81a25cc_db37_460c_8adb_9cfd35690280.slice/crio-ef45555f97030ea557a8b9707921ab23f82f806e9a1dfec6c800033a471aedb9 WatchSource:0}: Error finding container ef45555f97030ea557a8b9707921ab23f82f806e9a1dfec6c800033a471aedb9: Status 404 returned error can't find the container with id ef45555f97030ea557a8b9707921ab23f82f806e9a1dfec6c800033a471aedb9 Nov 03 21:07:36.033885 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:36.033850 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:07:36.033980 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:36.033924 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:07:36.033980 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:36.033956 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:07:36.130830 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:36.130791 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" event={"ID":"e81a25cc-db37-460c-8adb-9cfd35690280","Type":"ContainerStarted","Data":"0f698d270f80b858eea9d495c84f186e363e56702146cfe3d1d3d8535a1bfd22"} Nov 03 21:07:36.130997 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:36.130864 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" event={"ID":"e81a25cc-db37-460c-8adb-9cfd35690280","Type":"ContainerStarted","Data":"ef45555f97030ea557a8b9707921ab23f82f806e9a1dfec6c800033a471aedb9"} Nov 03 21:07:36.238865 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:36.238824 5615 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 21:07:36.239317 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:36.238896 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" podUID="7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 21:07:36.253525 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:36.253456 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" podStartSLOduration=1.253434978 podStartE2EDuration="1.253434978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:36.250025371 +0000 UTC m=+1150.717804094" watchObservedRunningTime="2026-11-03 21:07:36.253434978 +0000 UTC m=+1150.721213682" Nov 03 21:07:36.573191 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:36.573156 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:37.242102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:37.242033 5615 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 21:07:37.242654 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:37.242125 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" podUID="7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 21:07:37.574557 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:37.574471 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 21:07:37.574733 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:37.574563 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" podUID="e81a25cc-db37-460c-8adb-9cfd35690280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 21:07:38.240540 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:38.240497 5615 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 21:07:38.240848 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:38.240813 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" podUID="7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 21:07:38.574037 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:38.573954 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 21:07:38.574037 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:38.574028 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" podUID="e81a25cc-db37-460c-8adb-9cfd35690280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 21:07:39.244915 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:39.243496 5615 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 21:07:39.244915 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:39.243583 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" podUID="7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 21:07:39.573679 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:39.573583 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 21:07:39.573679 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:39.573643 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" podUID="e81a25cc-db37-460c-8adb-9cfd35690280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 21:07:40.238214 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:40.238173 5615 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 21:07:40.238602 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:40.238240 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" podUID="7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 21:07:40.574220 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:40.574122 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 21:07:40.574220 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:40.574175 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" podUID="e81a25cc-db37-460c-8adb-9cfd35690280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 21:07:40.635463 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:07:40.635388 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:07:41.242338 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:41.242308 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:41.242716 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:41.242680 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:41.243494 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:41.243477 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:07:41.573576 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:41.573470 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 21:07:41.573576 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:41.573536 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" podUID="e81a25cc-db37-460c-8adb-9cfd35690280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 21:07:42.573885 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:42.573853 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 21:07:42.574329 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:42.573904 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" podUID="e81a25cc-db37-460c-8adb-9cfd35690280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 21:07:43.577602 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:43.577568 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:43.578031 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:43.577823 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:43.578494 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:43.578476 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:07:51.242888 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.242853 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk"] Nov 03 21:07:51.246538 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.246515 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:51.434342 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.434294 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr7gc\" (UniqueName: \"kubernetes.io/projected/c34e4dc8-c666-4cf5-9a18-7fafb9607383-kube-api-access-fr7gc\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:51.434535 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.434398 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:51.434613 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.434532 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:51.434819 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.434653 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:51.434930 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.434885 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:51.434992 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.434957 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:51.435050 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.435024 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:51.435157 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.435141 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:51.435225 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.435185 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:51.438033 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.438007 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk"] Nov 03 21:07:51.536636 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.536550 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:51.536636 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.536608 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:51.536886 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.536645 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:51.536886 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.536695 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:51.536886 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.536828 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:51.537055 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.536897 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:51.537055 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.536946 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:51.537055 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.536989 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:51.537055 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.537040 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr7gc\" (UniqueName: \"kubernetes.io/projected/c34e4dc8-c666-4cf5-9a18-7fafb9607383-kube-api-access-fr7gc\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:51.537312 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.537143 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:51.537312 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.537248 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:51.537421 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.537395 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:51.537471 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.537449 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:51.537827 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.537800 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:51.539050 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.539027 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:51.539213 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.539197 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:51.595119 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.595061 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:51.595400 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.595378 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr7gc\" (UniqueName: \"kubernetes.io/projected/c34e4dc8-c666-4cf5-9a18-7fafb9607383-kube-api-access-fr7gc\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:51.861794 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:51.861701 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:52.283754 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:52.283723 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk"] Nov 03 21:07:52.285483 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:07:52.285454 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc34e4dc8_c666_4cf5_9a18_7fafb9607383.slice/crio-b83f6f020cb75a0dcbadfcb0279e5677d341f8e04ace39428fabb276f30e5199 WatchSource:0}: Error finding container b83f6f020cb75a0dcbadfcb0279e5677d341f8e04ace39428fabb276f30e5199: Status 404 returned error can't find the container with id b83f6f020cb75a0dcbadfcb0279e5677d341f8e04ace39428fabb276f30e5199 Nov 03 21:07:52.288211 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:52.288178 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:07:52.288290 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:52.288247 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:07:52.288290 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:52.288275 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:07:52.293577 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:52.293550 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" event={"ID":"c34e4dc8-c666-4cf5-9a18-7fafb9607383","Type":"ContainerStarted","Data":"b83f6f020cb75a0dcbadfcb0279e5677d341f8e04ace39428fabb276f30e5199"} Nov 03 21:07:53.327414 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:53.327373 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" event={"ID":"c34e4dc8-c666-4cf5-9a18-7fafb9607383","Type":"ContainerStarted","Data":"d85831cf86d0d2f19fbb138b983e13c980a884854f41b19c840e2503d10c6de2"} Nov 03 21:07:53.490152 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:53.490101 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" podStartSLOduration=2.490088051 podStartE2EDuration="2.490088051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:53.489401949 +0000 UTC m=+1167.957180672" watchObservedRunningTime="2026-11-03 21:07:53.490088051 +0000 UTC m=+1167.957866764" Nov 03 21:07:53.862055 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:53.862017 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:53.863630 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:53.863603 5615 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 21:07:53.863756 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:53.863650 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" podUID="c34e4dc8-c666-4cf5-9a18-7fafb9607383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 21:07:54.862320 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:54.862275 5615 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 21:07:54.862810 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:54.862350 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" podUID="c34e4dc8-c666-4cf5-9a18-7fafb9607383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 21:07:55.862536 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:55.862497 5615 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 21:07:55.862946 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:55.862562 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" podUID="c34e4dc8-c666-4cf5-9a18-7fafb9607383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 21:07:56.863204 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:56.863166 5615 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 21:07:56.863619 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:56.863232 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" podUID="c34e4dc8-c666-4cf5-9a18-7fafb9607383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 21:07:57.862191 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:57.862159 5615 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 21:07:57.862350 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:57.862214 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" podUID="c34e4dc8-c666-4cf5-9a18-7fafb9607383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 21:07:58.862510 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:58.862467 5615 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 21:07:58.862926 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:58.862525 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" podUID="c34e4dc8-c666-4cf5-9a18-7fafb9607383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 21:07:59.867652 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:59.867609 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:59.868439 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:59.868411 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:07:59.869017 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:07:59.868992 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:08:10.619380 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:08:10.619296 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:08:16.794103 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:16.792394 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" podUID="a89b0627-f993-4003-ac8b-fc449b44d9d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:08:20.595176 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.595142 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp"] Nov 03 21:08:20.598580 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.598558 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:20.674627 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.674589 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:20.674808 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.674716 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:20.674808 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.674763 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5n7n\" (UniqueName: \"kubernetes.io/projected/c7b4be3a-5908-4a2f-89b4-54308db81b2d-kube-api-access-b5n7n\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:20.674928 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.674884 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:20.674983 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.674948 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:20.675102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.675051 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:20.675233 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.675120 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:20.675233 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.675150 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:20.675325 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.675263 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:20.700248 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.700214 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp"] Nov 03 21:08:20.776210 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.776167 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:20.776210 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.776212 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:20.776446 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.776252 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:20.776446 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.776289 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:20.776446 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.776344 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:20.776446 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.776376 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5n7n\" (UniqueName: \"kubernetes.io/projected/c7b4be3a-5908-4a2f-89b4-54308db81b2d-kube-api-access-b5n7n\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:20.776446 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.776423 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:20.776719 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.776463 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:20.776719 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.776519 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:20.776868 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.776794 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:20.776868 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.776820 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:20.776981 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.776926 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:20.777038 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.777017 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:20.777314 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.777291 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:20.779939 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.779919 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:20.780191 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.780174 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:20.835155 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.835122 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5n7n\" (UniqueName: \"kubernetes.io/projected/c7b4be3a-5908-4a2f-89b4-54308db81b2d-kube-api-access-b5n7n\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:20.835385 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.835128 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:20.912539 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:20.912450 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:21.329784 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:21.329759 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp"] Nov 03 21:08:21.332271 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:08:21.332241 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b4be3a_5908_4a2f_89b4_54308db81b2d.slice/crio-7578bee8bccc123ffe84b11963038db0862572b9f063b6e253cf15ebe62eef7c WatchSource:0}: Error finding container 7578bee8bccc123ffe84b11963038db0862572b9f063b6e253cf15ebe62eef7c: Status 404 returned error can't find the container with id 7578bee8bccc123ffe84b11963038db0862572b9f063b6e253cf15ebe62eef7c Nov 03 21:08:21.334627 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:21.334591 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:08:21.334714 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:21.334657 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:08:21.334714 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:21.334683 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:08:21.733245 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:21.733214 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" event={"ID":"c7b4be3a-5908-4a2f-89b4-54308db81b2d","Type":"ContainerStarted","Data":"30ba6f47b14fef29588ded4d27eb35236a0c6644e9e69e453291789680878bcd"} Nov 03 21:08:21.733644 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:21.733276 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" event={"ID":"c7b4be3a-5908-4a2f-89b4-54308db81b2d","Type":"ContainerStarted","Data":"7578bee8bccc123ffe84b11963038db0862572b9f063b6e253cf15ebe62eef7c"} Nov 03 21:08:21.849274 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:21.849216 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" podStartSLOduration=1.849201251 podStartE2EDuration="1.849201251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:21.847102518 +0000 UTC m=+1196.314881244" watchObservedRunningTime="2026-11-03 21:08:21.849201251 +0000 UTC m=+1196.316979973" Nov 03 21:08:21.912952 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:21.912914 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:22.924316 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:22.924272 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 21:08:22.924838 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:22.924340 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" podUID="c7b4be3a-5908-4a2f-89b4-54308db81b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 21:08:23.916540 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:23.916498 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 21:08:23.916852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:23.916817 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" podUID="c7b4be3a-5908-4a2f-89b4-54308db81b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 21:08:24.916557 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:24.916516 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 21:08:24.917212 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:24.917180 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" podUID="c7b4be3a-5908-4a2f-89b4-54308db81b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 21:08:25.919566 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:25.919519 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 21:08:25.920203 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:25.920170 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" podUID="c7b4be3a-5908-4a2f-89b4-54308db81b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 21:08:26.919394 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:26.919357 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 21:08:26.919682 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:26.919651 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" podUID="c7b4be3a-5908-4a2f-89b4-54308db81b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 21:08:27.917657 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:27.917609 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 21:08:27.917971 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:27.917924 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" podUID="c7b4be3a-5908-4a2f-89b4-54308db81b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 21:08:28.916099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:28.914483 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 21:08:28.916099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:28.914560 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" podUID="c7b4be3a-5908-4a2f-89b4-54308db81b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 21:08:30.747098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:30.741989 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:30.747098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:30.746560 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:30.919935 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:30.919896 5615 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:08:30.940099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:30.934486 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:08:30.940099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:30.937404 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" podUID="c34e4dc8-c666-4cf5-9a18-7fafb9607383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:08:36.223950 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.223906 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229"] Nov 03 21:08:36.229840 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.229815 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:36.281642 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.281532 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:36.282008 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.281980 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:36.293726 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.293685 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:36.294160 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.294133 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:36.294388 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.294369 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:36.294577 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.294561 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:36.294881 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.294860 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:36.308259 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.308215 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn7nl\" (UniqueName: \"kubernetes.io/projected/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-kube-api-access-tn7nl\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:36.308494 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.308475 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:36.331098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.322138 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229"] Nov 03 21:08:36.409702 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.409662 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:36.409883 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.409713 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:36.409883 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.409745 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:36.409883 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.409767 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:36.409883 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.409814 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn7nl\" (UniqueName: \"kubernetes.io/projected/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-kube-api-access-tn7nl\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:36.409883 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.409847 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:36.410202 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.409928 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:36.410202 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.410021 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:36.410202 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.410058 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:36.410202 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.410168 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:36.410382 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.410207 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:36.410382 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.410277 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:36.410489 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.410442 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:36.410584 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.410566 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:36.412296 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.412273 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:36.412561 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.412542 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:36.472158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.472129 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:36.472430 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.472403 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn7nl\" (UniqueName: \"kubernetes.io/projected/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-kube-api-access-tn7nl\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:36.545624 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.545583 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:36.974624 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.974593 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229"] Nov 03 21:08:36.976068 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:08:36.976035 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0bf7948_0c3d_4cae_bd74_fa0f85d51573.slice/crio-73fc7079035888199088ac03f20e0661cc4f278a9678d05392ebc915aaed4c0c WatchSource:0}: Error finding container 73fc7079035888199088ac03f20e0661cc4f278a9678d05392ebc915aaed4c0c: Status 404 returned error can't find the container with id 73fc7079035888199088ac03f20e0661cc4f278a9678d05392ebc915aaed4c0c Nov 03 21:08:36.978897 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.978859 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:08:36.978983 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.978928 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:08:36.978983 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:36.978957 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:08:37.002225 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:37.002195 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" event={"ID":"a0bf7948-0c3d-4cae-bd74-fa0f85d51573","Type":"ContainerStarted","Data":"73fc7079035888199088ac03f20e0661cc4f278a9678d05392ebc915aaed4c0c"} Nov 03 21:08:38.027822 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:38.027782 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" event={"ID":"a0bf7948-0c3d-4cae-bd74-fa0f85d51573","Type":"ContainerStarted","Data":"a0e2c4272ebab31b5e80cd379d1d2a8282958174682321844137f2bb1065bd3a"} Nov 03 21:08:38.191695 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:38.191640 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" podStartSLOduration=2.191622829 podStartE2EDuration="2.191622829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:38.190158564 +0000 UTC m=+1212.657937287" watchObservedRunningTime="2026-11-03 21:08:38.191622829 +0000 UTC m=+1212.659401551" Nov 03 21:08:38.546126 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:38.546059 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:38.547557 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:38.547529 5615 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 21:08:38.547704 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:38.547583 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" podUID="a0bf7948-0c3d-4cae-bd74-fa0f85d51573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 21:08:39.546137 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:39.546057 5615 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 21:08:39.546536 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:39.546159 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" podUID="a0bf7948-0c3d-4cae-bd74-fa0f85d51573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 21:08:40.546154 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:40.546117 5615 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 21:08:40.546552 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:40.546172 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" podUID="a0bf7948-0c3d-4cae-bd74-fa0f85d51573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 21:08:40.646281 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:08:40.646239 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:08:41.546469 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:41.546428 5615 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 21:08:41.546873 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:41.546485 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" podUID="a0bf7948-0c3d-4cae-bd74-fa0f85d51573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 21:08:42.546161 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:42.546123 5615 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 21:08:42.546343 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:42.546181 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" podUID="a0bf7948-0c3d-4cae-bd74-fa0f85d51573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 21:08:43.546692 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:43.546191 5615 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 21:08:43.546692 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:43.546253 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" podUID="a0bf7948-0c3d-4cae-bd74-fa0f85d51573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 21:08:44.559108 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:44.554446 5615 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 21:08:44.559108 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:44.554527 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" podUID="a0bf7948-0c3d-4cae-bd74-fa0f85d51573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 21:08:46.590236 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:46.590203 5615 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:08:46.590713 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:46.590262 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" podUID="a0bf7948-0c3d-4cae-bd74-fa0f85d51573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:08:47.082553 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:47.082517 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:47.082941 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:47.082919 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:47.084399 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:47.084375 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:08:48.568559 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.568433 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6"] Nov 03 21:08:48.574586 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.574477 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:48.620771 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.620739 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh"] Nov 03 21:08:48.640017 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.639983 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:48.640870 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.640730 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6"] Nov 03 21:08:48.640993 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.640925 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh"] Nov 03 21:08:48.741451 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.741397 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:48.741451 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.741453 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:48.741719 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.741543 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5382faa-4094-4acf-a7e8-4538976acfb8-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:48.741719 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.741580 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:48.741719 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.741602 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5382faa-4094-4acf-a7e8-4538976acfb8-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:48.741719 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.741616 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:48.741719 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.741676 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrr4t\" (UniqueName: \"kubernetes.io/projected/eefda059-ea79-40bb-9304-12ab35450ab6-kube-api-access-qrr4t\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:48.741999 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.741788 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5382faa-4094-4acf-a7e8-4538976acfb8-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:48.741999 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.741809 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eefda059-ea79-40bb-9304-12ab35450ab6-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:48.741999 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.741887 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t76h\" (UniqueName: \"kubernetes.io/projected/f5382faa-4094-4acf-a7e8-4538976acfb8-kube-api-access-9t76h\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:48.741999 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.741926 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:48.742322 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.742011 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:48.742322 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.742066 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eefda059-ea79-40bb-9304-12ab35450ab6-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:48.742322 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.742160 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eefda059-ea79-40bb-9304-12ab35450ab6-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:48.742322 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.742188 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:48.742322 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.742204 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:48.742322 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.742300 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:48.742627 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.742325 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:48.843941 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.843853 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:48.843941 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.843903 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:48.843941 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.843932 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:48.844285 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.843947 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:48.844285 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.843963 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5382faa-4094-4acf-a7e8-4538976acfb8-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:48.844285 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.843984 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:48.844285 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.844106 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5382faa-4094-4acf-a7e8-4538976acfb8-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:48.844285 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.844144 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:48.844285 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.844168 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrr4t\" (UniqueName: \"kubernetes.io/projected/eefda059-ea79-40bb-9304-12ab35450ab6-kube-api-access-qrr4t\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:48.844285 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.844207 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5382faa-4094-4acf-a7e8-4538976acfb8-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:48.844285 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.844230 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eefda059-ea79-40bb-9304-12ab35450ab6-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:48.844694 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.844305 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t76h\" (UniqueName: \"kubernetes.io/projected/f5382faa-4094-4acf-a7e8-4538976acfb8-kube-api-access-9t76h\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:48.844694 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.844373 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:48.844694 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.844439 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:48.844694 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.844457 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:48.844694 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.844443 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:48.844694 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.844520 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:48.844694 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.844539 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eefda059-ea79-40bb-9304-12ab35450ab6-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:48.844694 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.844646 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eefda059-ea79-40bb-9304-12ab35450ab6-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:48.844694 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.844697 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:48.845232 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.844721 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:48.845232 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.844770 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:48.845232 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.845090 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:48.845232 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.845190 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:48.845232 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.845213 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5382faa-4094-4acf-a7e8-4538976acfb8-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:48.845504 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.845374 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:48.845504 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.845404 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:48.845857 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.845833 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eefda059-ea79-40bb-9304-12ab35450ab6-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:48.846882 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.846773 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:48.847006 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.846946 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:48.847460 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.847437 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5382faa-4094-4acf-a7e8-4538976acfb8-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:48.847798 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.847771 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eefda059-ea79-40bb-9304-12ab35450ab6-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:48.904187 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.904140 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eefda059-ea79-40bb-9304-12ab35450ab6-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:48.904408 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.904180 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5382faa-4094-4acf-a7e8-4538976acfb8-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:48.904408 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.904291 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrr4t\" (UniqueName: \"kubernetes.io/projected/eefda059-ea79-40bb-9304-12ab35450ab6-kube-api-access-qrr4t\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:48.904408 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.904309 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t76h\" (UniqueName: \"kubernetes.io/projected/f5382faa-4094-4acf-a7e8-4538976acfb8-kube-api-access-9t76h\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:48.946048 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.946012 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:48.959162 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:48.959133 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:49.374471 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:49.374445 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6"] Nov 03 21:08:49.375766 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:08:49.375738 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5382faa_4094_4acf_a7e8_4538976acfb8.slice/crio-4d2a7dc191b09887b78fc5549b143c5dee6405a47cfeeb33afdb1e09bc1b0d53 WatchSource:0}: Error finding container 4d2a7dc191b09887b78fc5549b143c5dee6405a47cfeeb33afdb1e09bc1b0d53: Status 404 returned error can't find the container with id 4d2a7dc191b09887b78fc5549b143c5dee6405a47cfeeb33afdb1e09bc1b0d53 Nov 03 21:08:49.377110 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:49.377066 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh"] Nov 03 21:08:49.378705 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:49.378670 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:08:49.378794 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:49.378747 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:08:49.378856 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:49.378792 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:08:49.379330 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:49.379242 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:08:49.379330 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:49.379297 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:08:49.379330 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:49.379327 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:08:50.192453 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:50.192411 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" event={"ID":"f5382faa-4094-4acf-a7e8-4538976acfb8","Type":"ContainerStarted","Data":"7a7af3bae7574fe7c19d468a021dc7bed857c50238383024db72ba5990a8ba80"} Nov 03 21:08:50.192952 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:50.192478 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" event={"ID":"f5382faa-4094-4acf-a7e8-4538976acfb8","Type":"ContainerStarted","Data":"4d2a7dc191b09887b78fc5549b143c5dee6405a47cfeeb33afdb1e09bc1b0d53"} Nov 03 21:08:50.193989 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:50.193965 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" event={"ID":"eefda059-ea79-40bb-9304-12ab35450ab6","Type":"ContainerStarted","Data":"084ff384b0a93912052fba6f528f15de3aa18b750e899028f80d1779249e0028"} Nov 03 21:08:50.194054 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:50.194007 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" event={"ID":"eefda059-ea79-40bb-9304-12ab35450ab6","Type":"ContainerStarted","Data":"651a866c50152f811689829e9e4f866457bc43ee82294fdaa025eb4d910f7af8"} Nov 03 21:08:50.429099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:50.414821 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" podStartSLOduration=2.414801432 podStartE2EDuration="2.414801432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:50.357317514 +0000 UTC m=+1224.825096237" watchObservedRunningTime="2026-11-03 21:08:50.414801432 +0000 UTC m=+1224.882580156" Nov 03 21:08:50.512097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:50.498118 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" podStartSLOduration=2.498098095 podStartE2EDuration="2.498098095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:50.495222873 +0000 UTC m=+1224.963001596" watchObservedRunningTime="2026-11-03 21:08:50.498098095 +0000 UTC m=+1224.965876816" Nov 03 21:08:50.949688 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:50.946433 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:50.955125 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:50.954672 5615 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 21:08:50.955125 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:50.954747 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" podUID="eefda059-ea79-40bb-9304-12ab35450ab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 21:08:50.964195 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:50.963840 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:50.964195 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:50.963983 5615 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 21:08:50.964195 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:50.964026 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" podUID="f5382faa-4094-4acf-a7e8-4538976acfb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 21:08:51.947099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:51.946577 5615 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 21:08:51.947099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:51.946643 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" podUID="eefda059-ea79-40bb-9304-12ab35450ab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 21:08:51.959662 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:51.959627 5615 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 21:08:51.959826 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:51.959681 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" podUID="f5382faa-4094-4acf-a7e8-4538976acfb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 21:08:52.950426 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:52.950387 5615 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 21:08:52.951367 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:52.951332 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" podUID="eefda059-ea79-40bb-9304-12ab35450ab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 21:08:52.959687 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:52.959634 5615 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 21:08:52.959811 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:52.959702 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" podUID="f5382faa-4094-4acf-a7e8-4538976acfb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 21:08:53.946645 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:53.946588 5615 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 21:08:53.946985 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:53.946925 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" podUID="eefda059-ea79-40bb-9304-12ab35450ab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 21:08:53.960155 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:53.960128 5615 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 21:08:53.960600 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:53.960577 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" podUID="f5382faa-4094-4acf-a7e8-4538976acfb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 21:08:54.947537 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:54.947499 5615 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 21:08:54.947837 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:54.947805 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" podUID="eefda059-ea79-40bb-9304-12ab35450ab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 21:08:54.959550 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:54.959520 5615 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 21:08:54.959764 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:54.959737 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" podUID="f5382faa-4094-4acf-a7e8-4538976acfb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 21:08:55.962781 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:55.962742 5615 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 21:08:55.963834 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:55.963788 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" podUID="eefda059-ea79-40bb-9304-12ab35450ab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 21:08:55.964034 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:55.963711 5615 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 21:08:55.964205 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:55.964182 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" podUID="f5382faa-4094-4acf-a7e8-4538976acfb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 21:08:56.951248 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:56.951213 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:56.951512 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:56.951493 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:56.952107 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:56.952084 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:08:56.964027 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:56.964003 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:56.964479 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:56.964391 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:08:56.964957 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:08:56.964939 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:09:09.792250 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:09.792193 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" podUID="a89b0627-f993-4003-ac8b-fc449b44d9d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:09:09.882291 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:09.882224 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" podUID="39c75c51-eff9-4cb8-88b9-618dbb2ba188" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:09:09.985205 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:09.985155 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" podUID="232c2524-2557-47c5-9d80-f6235972ca83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:09:10.233166 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:10.214208 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:09:10.233166 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:10.214305 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" podUID="884c8308-cf0b-496d-9f55-bb6563fe0238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:09:10.536160 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:10.534265 5615 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:09:10.536160 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:10.534327 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" podUID="13f7f450-acbe-4a27-aedc-4c3f43c942e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:09:10.879799 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:10.879691 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" podUID="a89b0627-f993-4003-ac8b-fc449b44d9d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:09:10.881567 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:10.881540 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" Nov 03 21:09:10.899998 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:10.885136 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" podUID="39c75c51-eff9-4cb8-88b9-618dbb2ba188" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 21:09:10.916199 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:10.910381 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" Nov 03 21:09:11.075480 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:11.053186 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" podUID="232c2524-2557-47c5-9d80-f6235972ca83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 21:09:11.075480 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:11.053576 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" Nov 03 21:09:11.715497 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:09:11.715434 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:09:11.917957 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:11.917914 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" podUID="a89b0627-f993-4003-ac8b-fc449b44d9d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 03 21:09:12.053552 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:12.053478 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" podUID="232c2524-2557-47c5-9d80-f6235972ca83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:09:12.071097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:12.067184 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" Nov 03 21:09:12.086424 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:12.086287 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" Nov 03 21:09:12.305308 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:12.305206 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" Nov 03 21:09:14.350770 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.350737 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np"] Nov 03 21:09:14.354917 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.354892 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:14.408103 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.407432 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-32f6e43c-openshift-default-dockercfg-cv7px\"" Nov 03 21:09:14.600098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.598525 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np"] Nov 03 21:09:14.600098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.599658 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-workload-socket\") pod \"e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:14.600098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.599751 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istio-podinfo\") pod \"e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:14.600098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.599780 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istio-token\") pod \"e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:14.605213 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.600889 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-credential-socket\") pod \"e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:14.605213 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.600954 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istio-envoy\") pod \"e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:14.605213 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.600978 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-workload-certs\") pod \"e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:14.605213 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.601031 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istio-data\") pod \"e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:14.605213 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.601105 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istiod-ca-cert\") pod \"e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:14.605213 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.601145 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9dwq\" (UniqueName: \"kubernetes.io/projected/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-kube-api-access-v9dwq\") pod \"e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:14.718112 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.715540 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istio-podinfo\") pod \"e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:14.718112 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.715601 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istio-token\") pod \"e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:14.718112 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.715747 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-credential-socket\") pod \"e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:14.718112 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.715798 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istio-envoy\") pod \"e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:14.718112 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.715822 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-workload-certs\") pod \"e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:14.718112 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.715875 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istio-data\") pod \"e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:14.718112 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.715986 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istiod-ca-cert\") pod \"e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:14.718112 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.716097 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9dwq\" (UniqueName: \"kubernetes.io/projected/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-kube-api-access-v9dwq\") pod \"e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:14.718112 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.716187 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-workload-socket\") pod \"e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:14.757749 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.739562 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istio-data\") pod \"e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:14.757749 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.753303 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istio-podinfo\") pod \"e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:14.757749 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.753322 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-credential-socket\") pod \"e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:14.757749 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.754430 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istiod-ca-cert\") pod \"e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:14.757749 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.755172 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-workload-socket\") pod \"e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:14.757749 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.755429 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-workload-certs\") pod \"e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:14.757749 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.756043 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istio-envoy\") pod \"e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:14.811354 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.811318 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istio-token\") pod \"e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:14.811883 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.811861 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9dwq\" (UniqueName: \"kubernetes.io/projected/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-kube-api-access-v9dwq\") pod \"e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:14.969344 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:14.969246 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:15.392382 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:15.392348 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np"] Nov 03 21:09:15.398178 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:15.398142 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:09:15.398267 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:15.398228 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:09:15.398320 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:15.398274 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:09:15.799175 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:15.799135 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" event={"ID":"c39084a3-cdf2-47b8-8aa1-b09415e6a90e","Type":"ContainerStarted","Data":"31513fd42aea31fc8770bcab3d3bb3b094e6646b731484d0fb5364935f52ea3b"} Nov 03 21:09:15.799353 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:15.799193 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" event={"ID":"c39084a3-cdf2-47b8-8aa1-b09415e6a90e","Type":"ContainerStarted","Data":"ef8397f2a820de0bc654271e9e86c9b4d09387d215b1f33d941ee7bd7a7883ca"} Nov 03 21:09:15.917852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:15.917794 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" podStartSLOduration=1.917779734 podStartE2EDuration="1.917779734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:09:15.914640498 +0000 UTC m=+1250.382419222" watchObservedRunningTime="2026-11-03 21:09:15.917779734 +0000 UTC m=+1250.385558457" Nov 03 21:09:15.969560 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:15.969522 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:16.975747 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:16.975712 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:17.812085 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:17.812054 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:17.813179 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:17.813158 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:09:22.787210 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:22.787173 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-kj8mx"] Nov 03 21:09:22.795026 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:22.794986 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-kj8mx" Nov 03 21:09:22.841098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:22.818680 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afb96de0-be6d-452d-bb92-5b72b3563cb6-plugins-config-volume\") pod \"payload-processing-e2e-mt-32f6e43c-fc4b7f84d-kj8mx\" (UID: \"afb96de0-be6d-452d-bb92-5b72b3563cb6\") " pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-kj8mx" Nov 03 21:09:22.841098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:22.818755 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bwlm\" (UniqueName: \"kubernetes.io/projected/afb96de0-be6d-452d-bb92-5b72b3563cb6-kube-api-access-2bwlm\") pod \"payload-processing-e2e-mt-32f6e43c-fc4b7f84d-kj8mx\" (UID: \"afb96de0-be6d-452d-bb92-5b72b3563cb6\") " pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-kj8mx" Nov 03 21:09:22.881102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:22.877484 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-kj8mx"] Nov 03 21:09:22.904100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:22.897397 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-32f6e43c\"" Nov 03 21:09:22.904100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:22.897558 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-32f6e43c-dockercfg-q28zk\"" Nov 03 21:09:22.932890 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:22.932720 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afb96de0-be6d-452d-bb92-5b72b3563cb6-plugins-config-volume\") pod \"payload-processing-e2e-mt-32f6e43c-fc4b7f84d-kj8mx\" (UID: \"afb96de0-be6d-452d-bb92-5b72b3563cb6\") " pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-kj8mx" Nov 03 21:09:22.932890 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:22.932788 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bwlm\" (UniqueName: \"kubernetes.io/projected/afb96de0-be6d-452d-bb92-5b72b3563cb6-kube-api-access-2bwlm\") pod \"payload-processing-e2e-mt-32f6e43c-fc4b7f84d-kj8mx\" (UID: \"afb96de0-be6d-452d-bb92-5b72b3563cb6\") " pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-kj8mx" Nov 03 21:09:22.942103 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:22.933806 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afb96de0-be6d-452d-bb92-5b72b3563cb6-plugins-config-volume\") pod \"payload-processing-e2e-mt-32f6e43c-fc4b7f84d-kj8mx\" (UID: \"afb96de0-be6d-452d-bb92-5b72b3563cb6\") " pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-kj8mx" Nov 03 21:09:23.011495 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:23.011383 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bwlm\" (UniqueName: \"kubernetes.io/projected/afb96de0-be6d-452d-bb92-5b72b3563cb6-kube-api-access-2bwlm\") pod \"payload-processing-e2e-mt-32f6e43c-fc4b7f84d-kj8mx\" (UID: \"afb96de0-be6d-452d-bb92-5b72b3563cb6\") " pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-kj8mx" Nov 03 21:09:23.150238 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:23.150150 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-kj8mx" Nov 03 21:09:23.521671 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:23.521637 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-kj8mx"] Nov 03 21:09:23.523786 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:09:23.523757 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb96de0_be6d_452d_bb92_5b72b3563cb6.slice/crio-e39496f8ec34ed6da1098429975d224de9fbc09a756621aa5419e2d5fa7a3618 WatchSource:0}: Error finding container e39496f8ec34ed6da1098429975d224de9fbc09a756621aa5419e2d5fa7a3618: Status 404 returned error can't find the container with id e39496f8ec34ed6da1098429975d224de9fbc09a756621aa5419e2d5fa7a3618 Nov 03 21:09:23.895024 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:23.894927 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-kj8mx" event={"ID":"afb96de0-be6d-452d-bb92-5b72b3563cb6","Type":"ContainerStarted","Data":"29d81ce462c869f58b307512a93c554ee348e65a8a11f7faaeead64c09ac808c"} Nov 03 21:09:23.895024 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:23.894983 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-kj8mx" event={"ID":"afb96de0-be6d-452d-bb92-5b72b3563cb6","Type":"ContainerStarted","Data":"e39496f8ec34ed6da1098429975d224de9fbc09a756621aa5419e2d5fa7a3618"} Nov 03 21:09:23.895663 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:23.895060 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-kj8mx" Nov 03 21:09:24.005115 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:24.005045 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-kj8mx" podStartSLOduration=2.005030657 podStartE2EDuration="2.005030657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:09:24.004137504 +0000 UTC m=+1258.471916228" watchObservedRunningTime="2026-11-03 21:09:24.005030657 +0000 UTC m=+1258.472809380" Nov 03 21:09:29.952100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:29.949105 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p"] Nov 03 21:09:29.952100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:29.949367 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" podUID="2a10a87e-4975-45cd-9f46-d9246dd3ae40" containerName="istio-proxy" containerID="cri-o://8ece380fe7f136457a470696900c7f1c7f159db587787bdc95416f236cf5c149" gracePeriod=30 Nov 03 21:09:34.933827 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:34.933796 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-kj8mx" Nov 03 21:09:35.548437 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:35.548405 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:09:35.650415 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:35.650318 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istio-podinfo\") pod \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " Nov 03 21:09:35.650415 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:35.650384 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istio-envoy\") pod \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " Nov 03 21:09:35.650649 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:35.650424 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istiod-ca-cert\") pod \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " Nov 03 21:09:35.650649 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:35.650466 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a10a87e-4975-45cd-9f46-d9246dd3ae40-kube-api-access-mw6bh\" (UniqueName: \"kubernetes.io/projected/2a10a87e-4975-45cd-9f46-d9246dd3ae40-kube-api-access-mw6bh\") pod \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " Nov 03 21:09:35.650649 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:35.650501 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istio-token\" (UniqueName: \"kubernetes.io/projected/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istio-token\") pod \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " Nov 03 21:09:35.650649 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:35.650526 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-workload-socket\") pod \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " Nov 03 21:09:35.650649 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:35.650583 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istio-data\") pod \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " Nov 03 21:09:35.650649 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:35.650607 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-credential-socket\") pod \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " Nov 03 21:09:35.650956 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:35.650671 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-workload-certs\") pod \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\" (UID: \"2a10a87e-4975-45cd-9f46-d9246dd3ae40\") " Nov 03 21:09:35.651143 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:35.651109 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istio-data" pod "2a10a87e-4975-45cd-9f46-d9246dd3ae40" (UID: "2a10a87e-4975-45cd-9f46-d9246dd3ae40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:35.651337 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:35.651206 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istiod-ca-cert" pod "2a10a87e-4975-45cd-9f46-d9246dd3ae40" (UID: "2a10a87e-4975-45cd-9f46-d9246dd3ae40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:09:35.651337 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:35.651292 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-workload-socket" pod "2a10a87e-4975-45cd-9f46-d9246dd3ae40" (UID: "2a10a87e-4975-45cd-9f46-d9246dd3ae40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:35.651529 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:35.651475 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-workload-certs" pod "2a10a87e-4975-45cd-9f46-d9246dd3ae40" (UID: "2a10a87e-4975-45cd-9f46-d9246dd3ae40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:35.651529 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:35.651492 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-credential-socket" pod "2a10a87e-4975-45cd-9f46-d9246dd3ae40" (UID: "2a10a87e-4975-45cd-9f46-d9246dd3ae40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:35.651529 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:35.651507 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:35.651529 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:35.651524 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:35.651736 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:35.651534 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:35.652882 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:35.652856 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istio-podinfo" pod "2a10a87e-4975-45cd-9f46-d9246dd3ae40" (UID: "2a10a87e-4975-45cd-9f46-d9246dd3ae40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:09:35.653134 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:35.653100 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istio-envoy" pod "2a10a87e-4975-45cd-9f46-d9246dd3ae40" (UID: "2a10a87e-4975-45cd-9f46-d9246dd3ae40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:35.653314 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:35.653298 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a10a87e-4975-45cd-9f46-d9246dd3ae40-kube-api-access-mw6bh" pod "2a10a87e-4975-45cd-9f46-d9246dd3ae40" (UID: "2a10a87e-4975-45cd-9f46-d9246dd3ae40"). InnerVolumeSpecName "kube-api-access-mw6bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:35.653476 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:35.653456 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istio-token" pod "2a10a87e-4975-45cd-9f46-d9246dd3ae40" (UID: "2a10a87e-4975-45cd-9f46-d9246dd3ae40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:35.753093 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:35.753040 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:35.753093 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:35.753091 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw6bh\" (UniqueName: \"kubernetes.io/projected/2a10a87e-4975-45cd-9f46-d9246dd3ae40-kube-api-access-mw6bh\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:35.753369 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:35.753106 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:35.753369 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:35.753122 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:35.753369 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:35.753136 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a10a87e-4975-45cd-9f46-d9246dd3ae40-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:35.753369 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:35.753151 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a10a87e-4975-45cd-9f46-d9246dd3ae40-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:36.284930 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:36.284890 5615 generic.go:356] "Generic (PLEG): container finished" podID="2a10a87e-4975-45cd-9f46-d9246dd3ae40" containerID="8ece380fe7f136457a470696900c7f1c7f159db587787bdc95416f236cf5c149" exitCode=0 Nov 03 21:09:36.285426 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:36.284948 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" Nov 03 21:09:36.285426 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:36.284956 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" event={"ID":"2a10a87e-4975-45cd-9f46-d9246dd3ae40","Type":"ContainerDied","Data":"8ece380fe7f136457a470696900c7f1c7f159db587787bdc95416f236cf5c149"} Nov 03 21:09:36.285426 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:36.285003 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p" event={"ID":"2a10a87e-4975-45cd-9f46-d9246dd3ae40","Type":"ContainerDied","Data":"22d72fb0f98ae0e640591ef84fc2ad613bb6d25be8e0989f2ab06d8750d45c82"} Nov 03 21:09:36.285426 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:36.285026 5615 scope.go:122] "RemoveContainer" containerID="8ece380fe7f136457a470696900c7f1c7f159db587787bdc95416f236cf5c149" Nov 03 21:09:36.295843 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:36.295817 5615 scope.go:122] "RemoveContainer" containerID="f986576b349e1b26761aab07bd273735ee57328db214e15ca51196de238ca8d7" Nov 03 21:09:36.304630 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:36.304606 5615 scope.go:122] "RemoveContainer" containerID="8ece380fe7f136457a470696900c7f1c7f159db587787bdc95416f236cf5c149" Nov 03 21:09:36.304881 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:09:36.304861 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ece380fe7f136457a470696900c7f1c7f159db587787bdc95416f236cf5c149\": container with ID starting with 8ece380fe7f136457a470696900c7f1c7f159db587787bdc95416f236cf5c149 not found: ID does not exist" containerID="8ece380fe7f136457a470696900c7f1c7f159db587787bdc95416f236cf5c149" Nov 03 21:09:36.304964 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:36.304894 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ece380fe7f136457a470696900c7f1c7f159db587787bdc95416f236cf5c149"} err="failed to get container status \"8ece380fe7f136457a470696900c7f1c7f159db587787bdc95416f236cf5c149\": rpc error: code = NotFound desc = could not find container \"8ece380fe7f136457a470696900c7f1c7f159db587787bdc95416f236cf5c149\": container with ID starting with 8ece380fe7f136457a470696900c7f1c7f159db587787bdc95416f236cf5c149 not found: ID does not exist" Nov 03 21:09:36.304964 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:36.304920 5615 scope.go:122] "RemoveContainer" containerID="f986576b349e1b26761aab07bd273735ee57328db214e15ca51196de238ca8d7" Nov 03 21:09:36.305177 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:09:36.305159 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f986576b349e1b26761aab07bd273735ee57328db214e15ca51196de238ca8d7\": container with ID starting with f986576b349e1b26761aab07bd273735ee57328db214e15ca51196de238ca8d7 not found: ID does not exist" containerID="f986576b349e1b26761aab07bd273735ee57328db214e15ca51196de238ca8d7" Nov 03 21:09:36.305226 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:36.305186 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f986576b349e1b26761aab07bd273735ee57328db214e15ca51196de238ca8d7"} err="failed to get container status \"f986576b349e1b26761aab07bd273735ee57328db214e15ca51196de238ca8d7\": rpc error: code = NotFound desc = could not find container \"f986576b349e1b26761aab07bd273735ee57328db214e15ca51196de238ca8d7\": container with ID starting with f986576b349e1b26761aab07bd273735ee57328db214e15ca51196de238ca8d7 not found: ID does not exist" Nov 03 21:09:36.446688 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:36.446643 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p"] Nov 03 21:09:36.494114 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:36.494064 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4bz5p"] Nov 03 21:09:37.652227 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:37.652191 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a10a87e-4975-45cd-9f46-d9246dd3ae40" path="/var/lib/kubelet/pods/2a10a87e-4975-45cd-9f46-d9246dd3ae40/volumes" Nov 03 21:09:40.619028 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:09:40.618993 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:09:46.490141 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.490108 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22"] Nov 03 21:09:46.490774 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.490760 5615 cpu_manager.go:414] "containerMap: removing container" podUID="2a10a87e-4975-45cd-9f46-d9246dd3ae40" containerName="istio-proxy" Nov 03 21:09:46.490816 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.490775 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a10a87e-4975-45cd-9f46-d9246dd3ae40" containerName="istio-proxy" Nov 03 21:09:46.490884 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.490874 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a10a87e-4975-45cd-9f46-d9246dd3ae40" containerName="istio-proxy" Nov 03 21:09:46.490918 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.490885 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a10a87e-4975-45cd-9f46-d9246dd3ae40" containerName="istio-proxy" Nov 03 21:09:46.533636 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.533604 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:46.661525 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.621977 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22"] Nov 03 21:09:46.663902 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.651800 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-145cc5-openshift-default-dockercfg-5s57x\"" Nov 03 21:09:46.664374 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.653531 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-istio-data\") pod \"e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:46.664806 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.664781 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-credential-socket\") pod \"e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:46.665456 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.665433 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02a77aa8-85a7-434b-9724-4e787da56d34-istio-podinfo\") pod \"e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:46.665767 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.665749 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-workload-certs\") pod \"e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:46.666061 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.666042 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02a77aa8-85a7-434b-9724-4e787da56d34-istiod-ca-cert\") pod \"e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:46.666585 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.666564 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02a77aa8-85a7-434b-9724-4e787da56d34-istio-token\") pod \"e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:46.667189 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.667063 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lktd5\" (UniqueName: \"kubernetes.io/projected/02a77aa8-85a7-434b-9724-4e787da56d34-kube-api-access-lktd5\") pod \"e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:46.667382 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.667355 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-workload-socket\") pod \"e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:46.667690 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.667671 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-istio-envoy\") pod \"e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:46.775152 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.768947 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02a77aa8-85a7-434b-9724-4e787da56d34-istio-podinfo\") pod \"e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:46.775152 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.769036 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-workload-certs\") pod \"e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:46.775152 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.769120 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02a77aa8-85a7-434b-9724-4e787da56d34-istiod-ca-cert\") pod \"e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:46.775152 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.769153 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02a77aa8-85a7-434b-9724-4e787da56d34-istio-token\") pod \"e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:46.775152 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.769231 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lktd5\" (UniqueName: \"kubernetes.io/projected/02a77aa8-85a7-434b-9724-4e787da56d34-kube-api-access-lktd5\") pod \"e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:46.775152 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.769276 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-workload-socket\") pod \"e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:46.775152 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.769347 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-istio-envoy\") pod \"e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:46.775152 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.769390 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-istio-data\") pod \"e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:46.775152 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.769431 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-credential-socket\") pod \"e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:46.775152 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.770059 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-credential-socket\") pod \"e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:46.775152 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.774935 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-workload-certs\") pod \"e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:46.804044 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.783849 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02a77aa8-85a7-434b-9724-4e787da56d34-istiod-ca-cert\") pod \"e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:46.804044 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.791234 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-workload-socket\") pod \"e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:46.804044 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.791512 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-istio-data\") pod \"e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:46.875061 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.875026 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02a77aa8-85a7-434b-9724-4e787da56d34-istio-token\") pod \"e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:46.943279 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.942015 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-istio-envoy\") pod \"e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:46.946678 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.946645 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02a77aa8-85a7-434b-9724-4e787da56d34-istio-podinfo\") pod \"e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:46.948812 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:46.948777 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lktd5\" (UniqueName: \"kubernetes.io/projected/02a77aa8-85a7-434b-9724-4e787da56d34-kube-api-access-lktd5\") pod \"e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:47.169601 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:47.169514 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:47.595087 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:47.594859 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22"] Nov 03 21:09:47.598522 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:09:47.598488 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a77aa8_85a7_434b_9724_4e787da56d34.slice/crio-25e647886071e31ef63f5264086462a7014433f42b4b28908a52551b873b2173 WatchSource:0}: Error finding container 25e647886071e31ef63f5264086462a7014433f42b4b28908a52551b873b2173: Status 404 returned error can't find the container with id 25e647886071e31ef63f5264086462a7014433f42b4b28908a52551b873b2173 Nov 03 21:09:47.601299 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:47.601252 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:09:47.601381 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:47.601353 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:09:47.601420 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:47.601404 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:09:47.804455 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:47.804421 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4"] Nov 03 21:09:47.805167 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:47.805145 5615 cpu_manager.go:414] "containerMap: removing container" podUID="2a10a87e-4975-45cd-9f46-d9246dd3ae40" containerName="istio-proxy" Nov 03 21:09:47.805167 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:47.805166 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a10a87e-4975-45cd-9f46-d9246dd3ae40" containerName="istio-proxy" Nov 03 21:09:47.808801 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:47.808742 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4" Nov 03 21:09:47.865205 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:47.863013 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-32f6e43c-serving-cert\"" Nov 03 21:09:47.866023 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:47.865988 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4"] Nov 03 21:09:47.917086 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:47.917037 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04594780-d2d0-4b95-85bf-db2578d8876f-maas-api-tls\") pod \"maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4\" (UID: \"04594780-d2d0-4b95-85bf-db2578d8876f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4" Nov 03 21:09:47.917272 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:47.917186 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04594780-d2d0-4b95-85bf-db2578d8876f-openshift-service-ca\") pod \"maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4\" (UID: \"04594780-d2d0-4b95-85bf-db2578d8876f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4" Nov 03 21:09:47.917339 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:47.917272 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04594780-d2d0-4b95-85bf-db2578d8876f-metrics-certs\") pod \"maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4\" (UID: \"04594780-d2d0-4b95-85bf-db2578d8876f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4" Nov 03 21:09:47.917442 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:47.917418 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z56n4\" (UniqueName: \"kubernetes.io/projected/04594780-d2d0-4b95-85bf-db2578d8876f-kube-api-access-z56n4\") pod \"maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4\" (UID: \"04594780-d2d0-4b95-85bf-db2578d8876f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4" Nov 03 21:09:48.019107 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:48.019045 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z56n4\" (UniqueName: \"kubernetes.io/projected/04594780-d2d0-4b95-85bf-db2578d8876f-kube-api-access-z56n4\") pod \"maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4\" (UID: \"04594780-d2d0-4b95-85bf-db2578d8876f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4" Nov 03 21:09:48.019291 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:48.019266 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04594780-d2d0-4b95-85bf-db2578d8876f-maas-api-tls\") pod \"maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4\" (UID: \"04594780-d2d0-4b95-85bf-db2578d8876f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4" Nov 03 21:09:48.019356 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:48.019342 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04594780-d2d0-4b95-85bf-db2578d8876f-openshift-service-ca\") pod \"maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4\" (UID: \"04594780-d2d0-4b95-85bf-db2578d8876f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4" Nov 03 21:09:48.019442 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:48.019426 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04594780-d2d0-4b95-85bf-db2578d8876f-metrics-certs\") pod \"maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4\" (UID: \"04594780-d2d0-4b95-85bf-db2578d8876f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4" Nov 03 21:09:48.019498 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:09:48.019432 5615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-serving-cert: secret "maas-api-e2e-mt-32f6e43c-serving-cert" not found Nov 03 21:09:48.019557 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:09:48.019506 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04594780-d2d0-4b95-85bf-db2578d8876f-maas-api-tls podName:04594780-d2d0-4b95-85bf-db2578d8876f nodeName:}" failed. No retries permitted until 2026-11-03 21:09:48.519484345 +0000 UTC m=+1282.987263060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/04594780-d2d0-4b95-85bf-db2578d8876f-maas-api-tls") pod "maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4" (UID: "04594780-d2d0-4b95-85bf-db2578d8876f") : secret "maas-api-e2e-mt-32f6e43c-serving-cert" not found Nov 03 21:09:48.020055 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:48.020034 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04594780-d2d0-4b95-85bf-db2578d8876f-openshift-service-ca\") pod \"maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4\" (UID: \"04594780-d2d0-4b95-85bf-db2578d8876f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4" Nov 03 21:09:48.022054 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:48.022027 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04594780-d2d0-4b95-85bf-db2578d8876f-metrics-certs\") pod \"maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4\" (UID: \"04594780-d2d0-4b95-85bf-db2578d8876f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4" Nov 03 21:09:48.098099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:48.085715 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z56n4\" (UniqueName: \"kubernetes.io/projected/04594780-d2d0-4b95-85bf-db2578d8876f-kube-api-access-z56n4\") pod \"maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4\" (UID: \"04594780-d2d0-4b95-85bf-db2578d8876f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4" Nov 03 21:09:48.422165 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:48.422127 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" event={"ID":"02a77aa8-85a7-434b-9724-4e787da56d34","Type":"ContainerStarted","Data":"146e57fd4b068f61ad45c53e0244223188aa84574882580d5240c4699b44f9d7"} Nov 03 21:09:48.422340 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:48.422184 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" event={"ID":"02a77aa8-85a7-434b-9724-4e787da56d34","Type":"ContainerStarted","Data":"25e647886071e31ef63f5264086462a7014433f42b4b28908a52551b873b2173"} Nov 03 21:09:48.529429 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:48.529390 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04594780-d2d0-4b95-85bf-db2578d8876f-maas-api-tls\") pod \"maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4\" (UID: \"04594780-d2d0-4b95-85bf-db2578d8876f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4" Nov 03 21:09:48.531820 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:48.531792 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04594780-d2d0-4b95-85bf-db2578d8876f-maas-api-tls\") pod \"maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4\" (UID: \"04594780-d2d0-4b95-85bf-db2578d8876f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4" Nov 03 21:09:48.540256 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:48.540207 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" podStartSLOduration=2.5401941040000002 podStartE2EDuration="2.540194104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:09:48.537487187 +0000 UTC m=+1283.005265899" watchObservedRunningTime="2026-11-03 21:09:48.540194104 +0000 UTC m=+1283.007972826" Nov 03 21:09:48.725168 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:48.724354 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4" Nov 03 21:09:49.125497 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:49.125458 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4"] Nov 03 21:09:49.127321 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:09:49.127285 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04594780_d2d0_4b95_85bf_db2578d8876f.slice/crio-5167c5edf7917cc7f4383cfe07b15df832ea2d3285723695f9171f402fba5260 WatchSource:0}: Error finding container 5167c5edf7917cc7f4383cfe07b15df832ea2d3285723695f9171f402fba5260: Status 404 returned error can't find the container with id 5167c5edf7917cc7f4383cfe07b15df832ea2d3285723695f9171f402fba5260 Nov 03 21:09:49.128895 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:49.128877 5615 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:09:49.170202 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:49.170167 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:49.175787 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:49.175756 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:49.435434 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:49.435341 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4" event={"ID":"04594780-d2d0-4b95-85bf-db2578d8876f","Type":"ContainerStarted","Data":"5167c5edf7917cc7f4383cfe07b15df832ea2d3285723695f9171f402fba5260"} Nov 03 21:09:49.435745 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:49.435717 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:49.436828 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:49.436806 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:09:50.442828 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:50.442736 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4" event={"ID":"04594780-d2d0-4b95-85bf-db2578d8876f","Type":"ContainerStarted","Data":"d5b54611fa75ce70c6fa1b2991cd76926cf22e7bb2b3122e49dc2cd6d8de9522"} Nov 03 21:09:50.443278 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:50.443027 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4" Nov 03 21:09:50.557347 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:50.557295 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4" podStartSLOduration=2.546641858 podStartE2EDuration="3.557279549s" totalImagesPullingTime="1.010637691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:09:50.555001219 +0000 UTC m=+1285.022779943" watchObservedRunningTime="2026-11-03 21:09:50.557279549 +0000 UTC m=+1285.025058272" Nov 03 21:09:56.454506 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:09:56.454477 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4" Nov 03 21:10:01.094245 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:01.094185 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" podUID="232c2524-2557-47c5-9d80-f6235972ca83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:10:01.324462 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:01.324308 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" podUID="39c75c51-eff9-4cb8-88b9-618dbb2ba188" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:10:09.711722 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:09.711684 5615 generic.go:356] "Generic (PLEG): container finished" podID="c39084a3-cdf2-47b8-8aa1-b09415e6a90e" containerID="31513fd42aea31fc8770bcab3d3bb3b094e6646b731484d0fb5364935f52ea3b" exitCode=0 Nov 03 21:10:09.712209 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:09.711751 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" event={"ID":"c39084a3-cdf2-47b8-8aa1-b09415e6a90e","Type":"ContainerDied","Data":"31513fd42aea31fc8770bcab3d3bb3b094e6646b731484d0fb5364935f52ea3b"} Nov 03 21:10:09.712209 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:09.712163 5615 scope.go:122] "RemoveContainer" containerID="31513fd42aea31fc8770bcab3d3bb3b094e6646b731484d0fb5364935f52ea3b" Nov 03 21:10:09.713505 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:09.713475 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:10:09.713572 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:09.713559 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:10:09.713609 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:09.713597 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:10:10.840429 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:10.839326 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" event={"ID":"c39084a3-cdf2-47b8-8aa1-b09415e6a90e","Type":"ContainerStarted","Data":"a0d42e39b268b810fa13e640e811f72567868c8f330fb2d2d3c1f286792680b1"} Nov 03 21:10:10.841780 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:10:10.841731 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:10:10.987690 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:10.987651 5615 patch_prober.go:29] interesting pod/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 21:10:10.987963 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:10.987932 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" podUID="c39084a3-cdf2-47b8-8aa1-b09415e6a90e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 21:10:10.988191 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:10.988163 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:10:11.969824 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:11.969790 5615 patch_prober.go:29] interesting pod/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 21:10:11.970367 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:11.969840 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" podUID="c39084a3-cdf2-47b8-8aa1-b09415e6a90e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 21:10:12.979855 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:12.979819 5615 patch_prober.go:29] interesting pod/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 21:10:12.988272 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:12.988226 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" podUID="c39084a3-cdf2-47b8-8aa1-b09415e6a90e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 21:10:13.074129 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:13.063516 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6dacc6-5d69b9ffff-dwd78"] Nov 03 21:10:13.137119 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:13.117252 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dacc6-5d69b9ffff-dwd78" Nov 03 21:10:13.137119 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:13.129680 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6dacc6-5d69b9ffff-dwd78"] Nov 03 21:10:13.176776 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:13.176745 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6dacc6-78c86d8fb4-9hmzf"] Nov 03 21:10:13.181276 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:13.181247 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6dacc6-78c86d8fb4-9hmzf" Nov 03 21:10:13.216363 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:13.216326 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-6dacc6\"" Nov 03 21:10:13.269386 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:13.226245 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-6dacc6-dockercfg-pc5kp\"" Nov 03 21:10:13.269612 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:13.252658 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af9e936f-0ebb-449b-b00b-0716f2525039-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6dacc6-5d69b9ffff-dwd78\" (UID: \"af9e936f-0ebb-449b-b00b-0716f2525039\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dacc6-5d69b9ffff-dwd78" Nov 03 21:10:13.269815 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:13.269778 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff023bfe-ed10-4ea6-bef1-bb220444bf00-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6dacc6-78c86d8fb4-9hmzf\" (UID: \"ff023bfe-ed10-4ea6-bef1-bb220444bf00\") " pod="openshift-ingress/payload-processing-e2e-api-a-6dacc6-78c86d8fb4-9hmzf" Nov 03 21:10:13.270176 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:13.270153 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqtm5\" (UniqueName: \"kubernetes.io/projected/af9e936f-0ebb-449b-b00b-0716f2525039-kube-api-access-jqtm5\") pod \"payload-pre-processing-e2e-api-a-6dacc6-5d69b9ffff-dwd78\" (UID: \"af9e936f-0ebb-449b-b00b-0716f2525039\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dacc6-5d69b9ffff-dwd78" Nov 03 21:10:13.270393 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:13.270372 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mfhh\" (UniqueName: \"kubernetes.io/projected/ff023bfe-ed10-4ea6-bef1-bb220444bf00-kube-api-access-5mfhh\") pod \"payload-processing-e2e-api-a-6dacc6-78c86d8fb4-9hmzf\" (UID: \"ff023bfe-ed10-4ea6-bef1-bb220444bf00\") " pod="openshift-ingress/payload-processing-e2e-api-a-6dacc6-78c86d8fb4-9hmzf" Nov 03 21:10:13.303285 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:13.303225 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6dacc6-78c86d8fb4-9hmzf"] Nov 03 21:10:13.434989 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:13.434936 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af9e936f-0ebb-449b-b00b-0716f2525039-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6dacc6-5d69b9ffff-dwd78\" (UID: \"af9e936f-0ebb-449b-b00b-0716f2525039\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dacc6-5d69b9ffff-dwd78" Nov 03 21:10:13.435207 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:13.435001 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff023bfe-ed10-4ea6-bef1-bb220444bf00-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6dacc6-78c86d8fb4-9hmzf\" (UID: \"ff023bfe-ed10-4ea6-bef1-bb220444bf00\") " pod="openshift-ingress/payload-processing-e2e-api-a-6dacc6-78c86d8fb4-9hmzf" Nov 03 21:10:13.444091 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:13.444029 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af9e936f-0ebb-449b-b00b-0716f2525039-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6dacc6-5d69b9ffff-dwd78\" (UID: \"af9e936f-0ebb-449b-b00b-0716f2525039\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dacc6-5d69b9ffff-dwd78" Nov 03 21:10:13.468138 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:13.468096 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff023bfe-ed10-4ea6-bef1-bb220444bf00-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6dacc6-78c86d8fb4-9hmzf\" (UID: \"ff023bfe-ed10-4ea6-bef1-bb220444bf00\") " pod="openshift-ingress/payload-processing-e2e-api-a-6dacc6-78c86d8fb4-9hmzf" Nov 03 21:10:13.468745 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:13.468203 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqtm5\" (UniqueName: \"kubernetes.io/projected/af9e936f-0ebb-449b-b00b-0716f2525039-kube-api-access-jqtm5\") pod \"payload-pre-processing-e2e-api-a-6dacc6-5d69b9ffff-dwd78\" (UID: \"af9e936f-0ebb-449b-b00b-0716f2525039\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dacc6-5d69b9ffff-dwd78" Nov 03 21:10:13.468745 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:13.468272 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mfhh\" (UniqueName: \"kubernetes.io/projected/ff023bfe-ed10-4ea6-bef1-bb220444bf00-kube-api-access-5mfhh\") pod \"payload-processing-e2e-api-a-6dacc6-78c86d8fb4-9hmzf\" (UID: \"ff023bfe-ed10-4ea6-bef1-bb220444bf00\") " pod="openshift-ingress/payload-processing-e2e-api-a-6dacc6-78c86d8fb4-9hmzf" Nov 03 21:10:13.541163 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:13.540999 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mfhh\" (UniqueName: \"kubernetes.io/projected/ff023bfe-ed10-4ea6-bef1-bb220444bf00-kube-api-access-5mfhh\") pod \"payload-processing-e2e-api-a-6dacc6-78c86d8fb4-9hmzf\" (UID: \"ff023bfe-ed10-4ea6-bef1-bb220444bf00\") " pod="openshift-ingress/payload-processing-e2e-api-a-6dacc6-78c86d8fb4-9hmzf" Nov 03 21:10:13.550184 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:13.550160 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqtm5\" (UniqueName: \"kubernetes.io/projected/af9e936f-0ebb-449b-b00b-0716f2525039-kube-api-access-jqtm5\") pod \"payload-pre-processing-e2e-api-a-6dacc6-5d69b9ffff-dwd78\" (UID: \"af9e936f-0ebb-449b-b00b-0716f2525039\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dacc6-5d69b9ffff-dwd78" Nov 03 21:10:13.607554 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:13.607507 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6dacc6-78c86d8fb4-9hmzf" Nov 03 21:10:13.827869 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:13.827528 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dacc6-5d69b9ffff-dwd78" Nov 03 21:10:13.970200 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:13.969860 5615 patch_prober.go:29] interesting pod/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 21:10:13.970200 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:13.969922 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" podUID="c39084a3-cdf2-47b8-8aa1-b09415e6a90e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 21:10:13.994815 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:10:13.994767 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff023bfe_ed10_4ea6_bef1_bb220444bf00.slice/crio-b2e0009603a795db533b4024afea4c9e598e9ef08aa58e14bfc29d888f238b59 WatchSource:0}: Error finding container b2e0009603a795db533b4024afea4c9e598e9ef08aa58e14bfc29d888f238b59: Status 404 returned error can't find the container with id b2e0009603a795db533b4024afea4c9e598e9ef08aa58e14bfc29d888f238b59 Nov 03 21:10:13.995740 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:13.995712 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6dacc6-78c86d8fb4-9hmzf"] Nov 03 21:10:14.344723 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:14.344679 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6dacc6-5d69b9ffff-dwd78"] Nov 03 21:10:14.970635 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:14.970597 5615 patch_prober.go:29] interesting pod/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 21:10:14.970913 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:14.970873 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" podUID="c39084a3-cdf2-47b8-8aa1-b09415e6a90e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 21:10:15.029122 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:15.019496 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6dacc6-78c86d8fb4-9hmzf" event={"ID":"ff023bfe-ed10-4ea6-bef1-bb220444bf00","Type":"ContainerStarted","Data":"57800c9abf3279ab463a195ade2d8c9a05d256bf77ac58cee3f10ae7d42951f7"} Nov 03 21:10:15.029122 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:15.019563 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6dacc6-78c86d8fb4-9hmzf" event={"ID":"ff023bfe-ed10-4ea6-bef1-bb220444bf00","Type":"ContainerStarted","Data":"b2e0009603a795db533b4024afea4c9e598e9ef08aa58e14bfc29d888f238b59"} Nov 03 21:10:15.029122 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:15.020596 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-6dacc6-78c86d8fb4-9hmzf" Nov 03 21:10:15.036213 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:15.036165 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dacc6-5d69b9ffff-dwd78" event={"ID":"af9e936f-0ebb-449b-b00b-0716f2525039","Type":"ContainerStarted","Data":"bbe7c825d14d906ac41669677b5b687b3c332e08ff85f0001ed987db8baff451"} Nov 03 21:10:15.036460 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:15.036243 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dacc6-5d69b9ffff-dwd78" event={"ID":"af9e936f-0ebb-449b-b00b-0716f2525039","Type":"ContainerStarted","Data":"d444aab9160bd680fcad52f63c6c48db508be17dabaf97006d346f3fbb39b049"} Nov 03 21:10:15.036838 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:15.036816 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dacc6-5d69b9ffff-dwd78" Nov 03 21:10:15.140273 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:15.140209 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-6dacc6-78c86d8fb4-9hmzf" podStartSLOduration=2.140192524 podStartE2EDuration="2.140192524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:15.13572834 +0000 UTC m=+1309.603507065" watchObservedRunningTime="2026-11-03 21:10:15.140192524 +0000 UTC m=+1309.607971246" Nov 03 21:10:15.249434 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:15.245831 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dacc6-5d69b9ffff-dwd78" podStartSLOduration=3.245812553 podStartE2EDuration="3.245812553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:15.245434565 +0000 UTC m=+1309.713213290" watchObservedRunningTime="2026-11-03 21:10:15.245812553 +0000 UTC m=+1309.713591276" Nov 03 21:10:15.970374 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:15.970335 5615 patch_prober.go:29] interesting pod/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 21:10:15.970544 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:15.970394 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" podUID="c39084a3-cdf2-47b8-8aa1-b09415e6a90e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 21:10:16.974222 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:16.974193 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:10:16.974712 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:16.974698 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:10:16.975463 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:16.975444 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:10:24.788257 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:24.788210 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:10:24.788727 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:24.788302 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" podUID="ebbedd3d-03f0-4518-b13b-fd88e4c633d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:10:26.046994 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:26.046954 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dacc6-5d69b9ffff-dwd78" Nov 03 21:10:27.051694 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:27.051661 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-6dacc6-78c86d8fb4-9hmzf" Nov 03 21:10:40.737326 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:10:40.737279 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:10:44.835581 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:44.835546 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw"] Nov 03 21:10:44.839673 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:44.839653 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw" Nov 03 21:10:44.938969 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:44.938929 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-6dacc6-serving-cert\"" Nov 03 21:10:44.942196 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:44.942164 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw"] Nov 03 21:10:44.945218 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:44.945190 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5baca77b-0b26-4236-ad3b-c0102e26b1eb-maas-api-tls\") pod \"maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw\" (UID: \"5baca77b-0b26-4236-ad3b-c0102e26b1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw" Nov 03 21:10:44.945381 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:44.945356 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5baca77b-0b26-4236-ad3b-c0102e26b1eb-metrics-certs\") pod \"maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw\" (UID: \"5baca77b-0b26-4236-ad3b-c0102e26b1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw" Nov 03 21:10:44.945455 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:44.945436 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvvwt\" (UniqueName: \"kubernetes.io/projected/5baca77b-0b26-4236-ad3b-c0102e26b1eb-kube-api-access-kvvwt\") pod \"maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw\" (UID: \"5baca77b-0b26-4236-ad3b-c0102e26b1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw" Nov 03 21:10:44.945635 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:44.945600 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5baca77b-0b26-4236-ad3b-c0102e26b1eb-openshift-service-ca\") pod \"maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw\" (UID: \"5baca77b-0b26-4236-ad3b-c0102e26b1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw" Nov 03 21:10:45.047348 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:45.047311 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5baca77b-0b26-4236-ad3b-c0102e26b1eb-metrics-certs\") pod \"maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw\" (UID: \"5baca77b-0b26-4236-ad3b-c0102e26b1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw" Nov 03 21:10:45.047551 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:45.047374 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvvwt\" (UniqueName: \"kubernetes.io/projected/5baca77b-0b26-4236-ad3b-c0102e26b1eb-kube-api-access-kvvwt\") pod \"maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw\" (UID: \"5baca77b-0b26-4236-ad3b-c0102e26b1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw" Nov 03 21:10:45.047551 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:45.047440 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5baca77b-0b26-4236-ad3b-c0102e26b1eb-openshift-service-ca\") pod \"maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw\" (UID: \"5baca77b-0b26-4236-ad3b-c0102e26b1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw" Nov 03 21:10:45.047679 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:45.047569 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5baca77b-0b26-4236-ad3b-c0102e26b1eb-maas-api-tls\") pod \"maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw\" (UID: \"5baca77b-0b26-4236-ad3b-c0102e26b1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw" Nov 03 21:10:45.047733 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:10:45.047712 5615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-serving-cert: secret "maas-api-e2e-api-a-6dacc6-serving-cert" not found Nov 03 21:10:45.047804 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:10:45.047793 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5baca77b-0b26-4236-ad3b-c0102e26b1eb-maas-api-tls podName:5baca77b-0b26-4236-ad3b-c0102e26b1eb nodeName:}" failed. No retries permitted until 2026-11-03 21:10:45.547772912 +0000 UTC m=+1340.015551625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5baca77b-0b26-4236-ad3b-c0102e26b1eb-maas-api-tls") pod "maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw" (UID: "5baca77b-0b26-4236-ad3b-c0102e26b1eb") : secret "maas-api-e2e-api-a-6dacc6-serving-cert" not found Nov 03 21:10:45.048397 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:45.048361 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5baca77b-0b26-4236-ad3b-c0102e26b1eb-openshift-service-ca\") pod \"maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw\" (UID: \"5baca77b-0b26-4236-ad3b-c0102e26b1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw" Nov 03 21:10:45.049999 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:45.049974 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5baca77b-0b26-4236-ad3b-c0102e26b1eb-metrics-certs\") pod \"maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw\" (UID: \"5baca77b-0b26-4236-ad3b-c0102e26b1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw" Nov 03 21:10:45.151734 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:45.151652 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvvwt\" (UniqueName: \"kubernetes.io/projected/5baca77b-0b26-4236-ad3b-c0102e26b1eb-kube-api-access-kvvwt\") pod \"maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw\" (UID: \"5baca77b-0b26-4236-ad3b-c0102e26b1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw" Nov 03 21:10:45.562707 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:45.562664 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5baca77b-0b26-4236-ad3b-c0102e26b1eb-maas-api-tls\") pod \"maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw\" (UID: \"5baca77b-0b26-4236-ad3b-c0102e26b1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw" Nov 03 21:10:45.565309 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:45.565278 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5baca77b-0b26-4236-ad3b-c0102e26b1eb-maas-api-tls\") pod \"maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw\" (UID: \"5baca77b-0b26-4236-ad3b-c0102e26b1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw" Nov 03 21:10:45.754271 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:45.754231 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw" Nov 03 21:10:46.133348 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:46.133318 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw"] Nov 03 21:10:46.135047 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:10:46.135015 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5baca77b_0b26_4236_ad3b_c0102e26b1eb.slice/crio-1ce78b99b9f27906cf96a55f27b1fbd61d6004ee6d570a87fabd068963de1734 WatchSource:0}: Error finding container 1ce78b99b9f27906cf96a55f27b1fbd61d6004ee6d570a87fabd068963de1734: Status 404 returned error can't find the container with id 1ce78b99b9f27906cf96a55f27b1fbd61d6004ee6d570a87fabd068963de1734 Nov 03 21:10:46.379766 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:46.379727 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw" event={"ID":"5baca77b-0b26-4236-ad3b-c0102e26b1eb","Type":"ContainerStarted","Data":"1ce78b99b9f27906cf96a55f27b1fbd61d6004ee6d570a87fabd068963de1734"} Nov 03 21:10:47.388340 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:47.388237 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw" event={"ID":"5baca77b-0b26-4236-ad3b-c0102e26b1eb","Type":"ContainerStarted","Data":"14de179b55d9ad9462d48d80e56a01f55cae8597295ab6d451fc4c5116c5f3ee"} Nov 03 21:10:47.388805 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:47.388383 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw" Nov 03 21:10:47.503484 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:47.503430 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw" podStartSLOduration=2.646300943 podStartE2EDuration="3.503413781s" totalImagesPullingTime="857.112838ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:47.500741626 +0000 UTC m=+1341.968520349" watchObservedRunningTime="2026-11-03 21:10:47.503413781 +0000 UTC m=+1341.971192503" Nov 03 21:10:48.396535 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:48.396499 5615 generic.go:356] "Generic (PLEG): container finished" podID="02a77aa8-85a7-434b-9724-4e787da56d34" containerID="146e57fd4b068f61ad45c53e0244223188aa84574882580d5240c4699b44f9d7" exitCode=0 Nov 03 21:10:48.397000 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:48.396574 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" event={"ID":"02a77aa8-85a7-434b-9724-4e787da56d34","Type":"ContainerDied","Data":"146e57fd4b068f61ad45c53e0244223188aa84574882580d5240c4699b44f9d7"} Nov 03 21:10:48.397069 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:48.397054 5615 scope.go:122] "RemoveContainer" containerID="146e57fd4b068f61ad45c53e0244223188aa84574882580d5240c4699b44f9d7" Nov 03 21:10:48.398394 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:48.398364 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:10:48.398512 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:48.398443 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:10:48.398512 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:48.398493 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:10:49.429708 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:49.429670 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" event={"ID":"02a77aa8-85a7-434b-9724-4e787da56d34","Type":"ContainerStarted","Data":"686332be9dc4bd010a266ec71e65074d5f7b4978f4fc8ec7697f7b105fadb006"} Nov 03 21:10:49.436301 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:49.436272 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:10:50.170167 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:50.170119 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:10:50.171272 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:50.171249 5615 patch_prober.go:29] interesting pod/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 21:10:50.171322 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:50.171299 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" podUID="02a77aa8-85a7-434b-9724-4e787da56d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 21:10:51.170226 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:51.170187 5615 patch_prober.go:29] interesting pod/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 21:10:51.170779 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:51.170255 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" podUID="02a77aa8-85a7-434b-9724-4e787da56d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 21:10:52.170193 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:52.170163 5615 patch_prober.go:29] interesting pod/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 21:10:52.170374 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:52.170217 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" podUID="02a77aa8-85a7-434b-9724-4e787da56d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 21:10:53.170521 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:53.170483 5615 patch_prober.go:29] interesting pod/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 21:10:53.170947 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:53.170549 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" podUID="02a77aa8-85a7-434b-9724-4e787da56d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 21:10:53.424028 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:53.423941 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw" Nov 03 21:10:54.170146 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:54.170102 5615 patch_prober.go:29] interesting pod/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 21:10:54.170326 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:54.170175 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" podUID="02a77aa8-85a7-434b-9724-4e787da56d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 21:10:55.174288 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:55.174256 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:10:55.175417 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:10:55.175394 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:11:10.619812 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:11:10.619727 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:11:19.699382 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.699343 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2"] Nov 03 21:11:19.707879 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.707840 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:19.760909 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.760876 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-6ad223-openshift-default-dockercfg-qjj85\"" Nov 03 21:11:19.787671 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.787631 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-workload-socket\") pod \"e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:19.787849 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.787793 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a388fecb-f964-46d3-a07e-4a87bf4c8981-istio-podinfo\") pod \"e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:19.787919 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.787905 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-istio-envoy\") pod \"e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:19.787967 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.787956 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-workload-certs\") pod \"e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:19.788005 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.787978 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a388fecb-f964-46d3-a07e-4a87bf4c8981-istiod-ca-cert\") pod \"e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:19.788058 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.788003 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-credential-socket\") pod \"e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:19.788248 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.788229 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a388fecb-f964-46d3-a07e-4a87bf4c8981-istio-token\") pod \"e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:19.788305 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.788280 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-istio-data\") pod \"e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:19.788454 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.788438 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77nfc\" (UniqueName: \"kubernetes.io/projected/a388fecb-f964-46d3-a07e-4a87bf4c8981-kube-api-access-77nfc\") pod \"e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:19.809933 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.809896 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2"] Nov 03 21:11:19.901450 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.893749 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-workload-socket\") pod \"e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:19.901450 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.893844 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a388fecb-f964-46d3-a07e-4a87bf4c8981-istio-podinfo\") pod \"e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:19.901450 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.893878 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-istio-envoy\") pod \"e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:19.901450 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.893932 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-workload-certs\") pod \"e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:19.901450 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.893956 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a388fecb-f964-46d3-a07e-4a87bf4c8981-istiod-ca-cert\") pod \"e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:19.901450 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.893991 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-credential-socket\") pod \"e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:19.901450 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.894171 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a388fecb-f964-46d3-a07e-4a87bf4c8981-istio-token\") pod \"e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:19.901450 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.894243 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-istio-data\") pod \"e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:19.901450 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.894367 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77nfc\" (UniqueName: \"kubernetes.io/projected/a388fecb-f964-46d3-a07e-4a87bf4c8981-kube-api-access-77nfc\") pod \"e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:19.901450 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.895255 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-workload-socket\") pod \"e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:19.901450 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.898418 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-credential-socket\") pod \"e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:19.901450 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.899703 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-istio-data\") pod \"e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:19.901450 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.900089 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-workload-certs\") pod \"e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:19.901450 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.900677 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a388fecb-f964-46d3-a07e-4a87bf4c8981-istiod-ca-cert\") pod \"e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:19.912856 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.912721 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-istio-envoy\") pod \"e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:19.926752 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.921382 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a388fecb-f964-46d3-a07e-4a87bf4c8981-istio-podinfo\") pod \"e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:19.964686 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.961243 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77nfc\" (UniqueName: \"kubernetes.io/projected/a388fecb-f964-46d3-a07e-4a87bf4c8981-kube-api-access-77nfc\") pod \"e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:19.964686 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:19.961580 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a388fecb-f964-46d3-a07e-4a87bf4c8981-istio-token\") pod \"e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:20.025099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:20.024357 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:20.535116 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:20.535060 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2"] Nov 03 21:11:20.542041 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:11:20.541985 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda388fecb_f964_46d3_a07e_4a87bf4c8981.slice/crio-28dc8ba9d6b188f18d285195987f6f60d11da60503d78d0b598166ba202ca79d WatchSource:0}: Error finding container 28dc8ba9d6b188f18d285195987f6f60d11da60503d78d0b598166ba202ca79d: Status 404 returned error can't find the container with id 28dc8ba9d6b188f18d285195987f6f60d11da60503d78d0b598166ba202ca79d Nov 03 21:11:20.548690 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:20.546288 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:11:20.548690 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:20.546388 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:11:20.548690 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:20.546430 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:11:20.702311 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:20.702270 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" event={"ID":"a388fecb-f964-46d3-a07e-4a87bf4c8981","Type":"ContainerStarted","Data":"1cfb6ca4f7f451d6f08c6e8d975894c03497943625786f17f4c97add819261b8"} Nov 03 21:11:20.703244 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:20.702330 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" event={"ID":"a388fecb-f964-46d3-a07e-4a87bf4c8981","Type":"ContainerStarted","Data":"28dc8ba9d6b188f18d285195987f6f60d11da60503d78d0b598166ba202ca79d"} Nov 03 21:11:21.036102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:21.025234 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:21.036102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:21.032792 5615 patch_prober.go:29] interesting pod/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 21:11:21.036102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:21.032955 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" podUID="a388fecb-f964-46d3-a07e-4a87bf4c8981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 21:11:22.030827 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:22.030792 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:22.205868 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:22.205813 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" podStartSLOduration=3.205797519 podStartE2EDuration="3.205797519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:11:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:11:20.839153716 +0000 UTC m=+1375.306932440" watchObservedRunningTime="2026-11-03 21:11:22.205797519 +0000 UTC m=+1376.673576241" Nov 03 21:11:22.715364 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:22.715333 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:22.716271 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:22.716247 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:11:27.242643 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:27.242602 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6ad223-56556d55ff-qnczx"] Nov 03 21:11:27.247137 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:27.247109 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ad223-56556d55ff-qnczx" Nov 03 21:11:27.296980 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:27.296947 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-6ad223\"" Nov 03 21:11:27.297249 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:27.297218 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-6ad223-dockercfg-l28s7\"" Nov 03 21:11:27.300783 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:27.300759 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6ad223-56556d55ff-qnczx"] Nov 03 21:11:27.352273 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:27.352224 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a07203b7-b82b-4329-9c71-f50ee6e77374-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6ad223-56556d55ff-qnczx\" (UID: \"a07203b7-b82b-4329-9c71-f50ee6e77374\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ad223-56556d55ff-qnczx" Nov 03 21:11:27.352479 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:27.352340 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz4kw\" (UniqueName: \"kubernetes.io/projected/a07203b7-b82b-4329-9c71-f50ee6e77374-kube-api-access-zz4kw\") pod \"payload-pre-processing-e2e-api-b-6ad223-56556d55ff-qnczx\" (UID: \"a07203b7-b82b-4329-9c71-f50ee6e77374\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ad223-56556d55ff-qnczx" Nov 03 21:11:27.469099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:27.467346 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz4kw\" (UniqueName: \"kubernetes.io/projected/a07203b7-b82b-4329-9c71-f50ee6e77374-kube-api-access-zz4kw\") pod \"payload-pre-processing-e2e-api-b-6ad223-56556d55ff-qnczx\" (UID: \"a07203b7-b82b-4329-9c71-f50ee6e77374\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ad223-56556d55ff-qnczx" Nov 03 21:11:27.469099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:27.467680 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a07203b7-b82b-4329-9c71-f50ee6e77374-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6ad223-56556d55ff-qnczx\" (UID: \"a07203b7-b82b-4329-9c71-f50ee6e77374\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ad223-56556d55ff-qnczx" Nov 03 21:11:27.469099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:27.468694 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a07203b7-b82b-4329-9c71-f50ee6e77374-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6ad223-56556d55ff-qnczx\" (UID: \"a07203b7-b82b-4329-9c71-f50ee6e77374\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ad223-56556d55ff-qnczx" Nov 03 21:11:27.563636 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:27.563536 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6ad223-577d847bf6-krjwf"] Nov 03 21:11:27.616356 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:27.615669 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6ad223-577d847bf6-krjwf" Nov 03 21:11:27.616622 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:27.616598 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6ad223-577d847bf6-krjwf"] Nov 03 21:11:27.621493 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:27.621463 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz4kw\" (UniqueName: \"kubernetes.io/projected/a07203b7-b82b-4329-9c71-f50ee6e77374-kube-api-access-zz4kw\") pod \"payload-pre-processing-e2e-api-b-6ad223-56556d55ff-qnczx\" (UID: \"a07203b7-b82b-4329-9c71-f50ee6e77374\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ad223-56556d55ff-qnczx" Nov 03 21:11:27.678037 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:27.677997 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589203b0-97f0-4e72-a1e3-091039e5d0ce-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6ad223-577d847bf6-krjwf\" (UID: \"589203b0-97f0-4e72-a1e3-091039e5d0ce\") " pod="openshift-ingress/payload-processing-e2e-api-b-6ad223-577d847bf6-krjwf" Nov 03 21:11:27.678350 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:27.678323 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd52s\" (UniqueName: \"kubernetes.io/projected/589203b0-97f0-4e72-a1e3-091039e5d0ce-kube-api-access-dd52s\") pod \"payload-processing-e2e-api-b-6ad223-577d847bf6-krjwf\" (UID: \"589203b0-97f0-4e72-a1e3-091039e5d0ce\") " pod="openshift-ingress/payload-processing-e2e-api-b-6ad223-577d847bf6-krjwf" Nov 03 21:11:27.779373 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:27.779340 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589203b0-97f0-4e72-a1e3-091039e5d0ce-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6ad223-577d847bf6-krjwf\" (UID: \"589203b0-97f0-4e72-a1e3-091039e5d0ce\") " pod="openshift-ingress/payload-processing-e2e-api-b-6ad223-577d847bf6-krjwf" Nov 03 21:11:27.779558 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:27.779538 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd52s\" (UniqueName: \"kubernetes.io/projected/589203b0-97f0-4e72-a1e3-091039e5d0ce-kube-api-access-dd52s\") pod \"payload-processing-e2e-api-b-6ad223-577d847bf6-krjwf\" (UID: \"589203b0-97f0-4e72-a1e3-091039e5d0ce\") " pod="openshift-ingress/payload-processing-e2e-api-b-6ad223-577d847bf6-krjwf" Nov 03 21:11:27.780192 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:27.780173 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589203b0-97f0-4e72-a1e3-091039e5d0ce-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6ad223-577d847bf6-krjwf\" (UID: \"589203b0-97f0-4e72-a1e3-091039e5d0ce\") " pod="openshift-ingress/payload-processing-e2e-api-b-6ad223-577d847bf6-krjwf" Nov 03 21:11:27.837705 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:27.837632 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd52s\" (UniqueName: \"kubernetes.io/projected/589203b0-97f0-4e72-a1e3-091039e5d0ce-kube-api-access-dd52s\") pod \"payload-processing-e2e-api-b-6ad223-577d847bf6-krjwf\" (UID: \"589203b0-97f0-4e72-a1e3-091039e5d0ce\") " pod="openshift-ingress/payload-processing-e2e-api-b-6ad223-577d847bf6-krjwf" Nov 03 21:11:27.862299 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:27.862263 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ad223-56556d55ff-qnczx" Nov 03 21:11:27.959427 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:27.959389 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6ad223-577d847bf6-krjwf" Nov 03 21:11:28.235797 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:28.235770 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6ad223-56556d55ff-qnczx"] Nov 03 21:11:28.236809 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:11:28.236778 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07203b7_b82b_4329_9c71_f50ee6e77374.slice/crio-06fa30c11f583132f7a3dce65f02805dc3532648cc7aa942b902ca47134b7eb2 WatchSource:0}: Error finding container 06fa30c11f583132f7a3dce65f02805dc3532648cc7aa942b902ca47134b7eb2: Status 404 returned error can't find the container with id 06fa30c11f583132f7a3dce65f02805dc3532648cc7aa942b902ca47134b7eb2 Nov 03 21:11:28.351011 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:28.350982 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6ad223-577d847bf6-krjwf"] Nov 03 21:11:28.351827 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:11:28.351799 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod589203b0_97f0_4e72_a1e3_091039e5d0ce.slice/crio-f02e8875ee61608748808c205bc4efb7ae8822c8239b389365b8ea987a0e6904 WatchSource:0}: Error finding container f02e8875ee61608748808c205bc4efb7ae8822c8239b389365b8ea987a0e6904: Status 404 returned error can't find the container with id f02e8875ee61608748808c205bc4efb7ae8822c8239b389365b8ea987a0e6904 Nov 03 21:11:28.771265 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:28.771224 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ad223-56556d55ff-qnczx" event={"ID":"a07203b7-b82b-4329-9c71-f50ee6e77374","Type":"ContainerStarted","Data":"7b4bcf59e7dccbd98e3641faf0b4ea2369387e829deb058e2f1827dda3314778"} Nov 03 21:11:28.771475 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:28.771294 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ad223-56556d55ff-qnczx" event={"ID":"a07203b7-b82b-4329-9c71-f50ee6e77374","Type":"ContainerStarted","Data":"06fa30c11f583132f7a3dce65f02805dc3532648cc7aa942b902ca47134b7eb2"} Nov 03 21:11:28.771475 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:28.771331 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ad223-56556d55ff-qnczx" Nov 03 21:11:28.772940 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:28.772914 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6ad223-577d847bf6-krjwf" event={"ID":"589203b0-97f0-4e72-a1e3-091039e5d0ce","Type":"ContainerStarted","Data":"5056c22ccdf4de15a414dae54d6fe19a533e00027ec07cae5d30f3d03301fb60"} Nov 03 21:11:28.773088 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:28.772964 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6ad223-577d847bf6-krjwf" event={"ID":"589203b0-97f0-4e72-a1e3-091039e5d0ce","Type":"ContainerStarted","Data":"f02e8875ee61608748808c205bc4efb7ae8822c8239b389365b8ea987a0e6904"} Nov 03 21:11:28.773088 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:28.772985 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-6ad223-577d847bf6-krjwf" Nov 03 21:11:28.883281 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:28.883223 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ad223-56556d55ff-qnczx" podStartSLOduration=1.883209671 podStartE2EDuration="1.883209671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:11:28.881187474 +0000 UTC m=+1383.348966202" watchObservedRunningTime="2026-11-03 21:11:28.883209671 +0000 UTC m=+1383.350988394" Nov 03 21:11:28.994211 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:28.994149 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-6ad223-577d847bf6-krjwf" podStartSLOduration=1.9941338320000002 podStartE2EDuration="1.994133832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:11:28.993345001 +0000 UTC m=+1383.461123726" watchObservedRunningTime="2026-11-03 21:11:28.994133832 +0000 UTC m=+1383.461912554" Nov 03 21:11:30.912098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:30.911315 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc"] Nov 03 21:11:30.927175 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:30.924447 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:30.983101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:30.982317 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:30.983101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:30.982407 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:30.983101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:30.982438 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:30.983101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:30.982535 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f336dd-0843-4263-90b8-25db7598d76f-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:30.983101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:30.982720 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:30.983101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:30.982756 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12f336dd-0843-4263-90b8-25db7598d76f-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:30.983101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:30.982783 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f336dd-0843-4263-90b8-25db7598d76f-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:30.983101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:30.982828 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:30.983101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:30.982879 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl6g7\" (UniqueName: \"kubernetes.io/projected/12f336dd-0843-4263-90b8-25db7598d76f-kube-api-access-vl6g7\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:31.038086 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:31.038023 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc"] Nov 03 21:11:31.039177 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:31.039151 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-e84357-openshift-default-dockercfg-t4w57\"" Nov 03 21:11:31.089845 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:31.089804 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:31.116041 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:31.116000 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12f336dd-0843-4263-90b8-25db7598d76f-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:31.116591 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:31.116564 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f336dd-0843-4263-90b8-25db7598d76f-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:31.116784 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:31.116765 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:31.116926 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:31.116910 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl6g7\" (UniqueName: \"kubernetes.io/projected/12f336dd-0843-4263-90b8-25db7598d76f-kube-api-access-vl6g7\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:31.117099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:31.117040 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:31.117289 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:31.117274 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:31.117587 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:31.117385 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:31.117838 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:31.117810 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f336dd-0843-4263-90b8-25db7598d76f-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:31.118066 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:31.118040 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:31.118411 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:31.118379 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:31.118649 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:31.118626 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:31.118733 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:31.117442 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f336dd-0843-4263-90b8-25db7598d76f-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:31.118876 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:31.118856 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:31.120594 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:31.120566 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:31.121611 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:31.121588 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f336dd-0843-4263-90b8-25db7598d76f-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:31.175135 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:31.175052 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12f336dd-0843-4263-90b8-25db7598d76f-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:31.175528 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:31.175497 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl6g7\" (UniqueName: \"kubernetes.io/projected/12f336dd-0843-4263-90b8-25db7598d76f-kube-api-access-vl6g7\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:31.285966 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:31.285928 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:31.710317 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:31.710279 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc"] Nov 03 21:11:31.719367 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:11:31.719332 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12f336dd_0843_4263_90b8_25db7598d76f.slice/crio-604190aa0f5693f46cb3fc9b719130c4191afc2bfbeae8e23735fdb1d0ad7472 WatchSource:0}: Error finding container 604190aa0f5693f46cb3fc9b719130c4191afc2bfbeae8e23735fdb1d0ad7472: Status 404 returned error can't find the container with id 604190aa0f5693f46cb3fc9b719130c4191afc2bfbeae8e23735fdb1d0ad7472 Nov 03 21:11:31.722100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:31.722047 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:11:31.722191 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:31.722138 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:11:31.722191 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:31.722172 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:11:31.796031 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:31.795993 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" event={"ID":"12f336dd-0843-4263-90b8-25db7598d76f","Type":"ContainerStarted","Data":"75abec1ae39fa222452f5820e7329ad71aa06457614847ff54c40ef13045c5cd"} Nov 03 21:11:31.796212 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:31.796061 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" event={"ID":"12f336dd-0843-4263-90b8-25db7598d76f","Type":"ContainerStarted","Data":"604190aa0f5693f46cb3fc9b719130c4191afc2bfbeae8e23735fdb1d0ad7472"} Nov 03 21:11:31.912845 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:31.912787 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" podStartSLOduration=1.912770685 podStartE2EDuration="1.912770685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:11:31.910379769 +0000 UTC m=+1386.378158505" watchObservedRunningTime="2026-11-03 21:11:31.912770685 +0000 UTC m=+1386.380549407" Nov 03 21:11:32.286696 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:32.286659 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:32.288150 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:32.288122 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 21:11:32.288278 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:32.288170 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" podUID="12f336dd-0843-4263-90b8-25db7598d76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 21:11:33.287147 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.287070 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 21:11:33.287547 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.287163 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" podUID="12f336dd-0843-4263-90b8-25db7598d76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 21:11:33.328388 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.328345 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p"] Nov 03 21:11:33.333391 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.333359 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:33.382953 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.382922 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-3e92e3-openshift-default-dockercfg-gkl2g\"" Nov 03 21:11:33.433116 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.431627 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p"] Nov 03 21:11:33.483371 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.483321 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istio-data\") pod \"e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:33.483758 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.483733 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-workload-certs\") pod \"e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:33.484013 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.483991 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-credential-socket\") pod \"e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:33.484222 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.484201 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9w47\" (UniqueName: \"kubernetes.io/projected/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-kube-api-access-r9w47\") pod \"e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:33.484481 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.484464 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istio-envoy\") pod \"e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:33.484627 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.484611 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istio-token\") pod \"e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:33.484801 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.484786 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istio-podinfo\") pod \"e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:33.484934 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.484920 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-workload-socket\") pod \"e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:33.485140 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.485124 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istiod-ca-cert\") pod \"e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:33.617180 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.588961 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istio-data\") pod \"e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:33.622000 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.621964 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-workload-certs\") pod \"e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:33.633605 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.633556 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-credential-socket\") pod \"e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:33.633856 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.633834 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9w47\" (UniqueName: \"kubernetes.io/projected/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-kube-api-access-r9w47\") pod \"e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:33.634044 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.634029 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istio-envoy\") pod \"e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:33.634197 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.634181 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istio-token\") pod \"e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:33.634378 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.634360 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istio-podinfo\") pod \"e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:33.634487 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.634473 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-workload-socket\") pod \"e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:33.634663 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.634648 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istiod-ca-cert\") pod \"e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:33.645108 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.635872 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istiod-ca-cert\") pod \"e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:33.645108 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.636557 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istio-data\") pod \"e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:33.645108 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.637258 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-workload-socket\") pod \"e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:33.645108 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.637511 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-credential-socket\") pod \"e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:33.645108 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.633287 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-workload-certs\") pod \"e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:33.732095 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.723279 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istio-envoy\") pod \"e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:33.732095 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.728322 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istio-podinfo\") pod \"e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:33.757100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.737817 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istio-token\") pod \"e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:33.757405 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.749000 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9w47\" (UniqueName: \"kubernetes.io/projected/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-kube-api-access-r9w47\") pod \"e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:33.952169 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:33.952082 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:34.286998 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:34.286947 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 21:11:34.287218 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:34.287024 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" podUID="12f336dd-0843-4263-90b8-25db7598d76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 21:11:34.428602 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:34.428563 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p"] Nov 03 21:11:34.429099 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:11:34.429053 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba344bf_b7b1_4bfe_8098_78073ecc5fb6.slice/crio-712d68890f262662e4ceca39d33453c56c75ef0c463640724113869a6a6552b8 WatchSource:0}: Error finding container 712d68890f262662e4ceca39d33453c56c75ef0c463640724113869a6a6552b8: Status 404 returned error can't find the container with id 712d68890f262662e4ceca39d33453c56c75ef0c463640724113869a6a6552b8 Nov 03 21:11:34.431744 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:34.431708 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:11:34.431833 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:34.431780 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:11:34.431833 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:34.431809 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:11:34.842982 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:34.842906 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" event={"ID":"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6","Type":"ContainerStarted","Data":"499d514c2ee1c87367e85a5b68ecf0f9c459b88c03fc834bc7b1265a8cd3ad79"} Nov 03 21:11:34.842982 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:34.842966 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" event={"ID":"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6","Type":"ContainerStarted","Data":"712d68890f262662e4ceca39d33453c56c75ef0c463640724113869a6a6552b8"} Nov 03 21:11:34.953068 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:34.953036 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:34.960723 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:34.960669 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" podStartSLOduration=1.96065475 podStartE2EDuration="1.96065475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:11:34.957533588 +0000 UTC m=+1389.425312312" watchObservedRunningTime="2026-11-03 21:11:34.96065475 +0000 UTC m=+1389.428433467" Nov 03 21:11:35.286584 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:35.286540 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 21:11:35.286800 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:35.286610 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" podUID="12f336dd-0843-4263-90b8-25db7598d76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 21:11:35.958418 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:35.958381 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:36.286532 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:36.286492 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 21:11:36.286739 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:36.286558 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" podUID="12f336dd-0843-4263-90b8-25db7598d76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 21:11:36.855384 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:36.855349 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:36.856230 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:36.856209 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:11:37.286356 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:37.286315 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 21:11:37.286853 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:37.286409 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" podUID="12f336dd-0843-4263-90b8-25db7598d76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 21:11:38.288089 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:38.288035 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 21:11:38.288728 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:38.288696 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" podUID="12f336dd-0843-4263-90b8-25db7598d76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 21:11:39.286831 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:39.286788 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 21:11:39.287033 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:39.286856 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" podUID="12f336dd-0843-4263-90b8-25db7598d76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 21:11:39.784107 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:39.784036 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ad223-56556d55ff-qnczx" Nov 03 21:11:39.784627 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:39.784281 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-6ad223-577d847bf6-krjwf" Nov 03 21:11:40.290999 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:40.290957 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:40.291341 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:40.291323 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:40.292147 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:40.292125 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:11:40.619605 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:11:40.619520 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:11:46.349858 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:46.349731 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3e92e3-85f886c5dd-c5pzb"] Nov 03 21:11:46.418418 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:46.418382 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3e92e3-85f886c5dd-c5pzb" Nov 03 21:11:46.488718 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:46.488642 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3e92e3-85f886c5dd-c5pzb"] Nov 03 21:11:46.494958 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:46.494873 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f16bde-5aa7-46f6-b801-68d84fa02ae3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3e92e3-85f886c5dd-c5pzb\" (UID: \"50f16bde-5aa7-46f6-b801-68d84fa02ae3\") " pod="openshift-ingress/payload-processing-e2e-api-a-3e92e3-85f886c5dd-c5pzb" Nov 03 21:11:46.495523 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:46.495471 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wns28\" (UniqueName: \"kubernetes.io/projected/50f16bde-5aa7-46f6-b801-68d84fa02ae3-kube-api-access-wns28\") pod \"payload-processing-e2e-api-a-3e92e3-85f886c5dd-c5pzb\" (UID: \"50f16bde-5aa7-46f6-b801-68d84fa02ae3\") " pod="openshift-ingress/payload-processing-e2e-api-a-3e92e3-85f886c5dd-c5pzb" Nov 03 21:11:46.535236 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:46.527382 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-3e92e3\"" Nov 03 21:11:46.550119 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:46.527911 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-3e92e3-dockercfg-xhkgj\"" Nov 03 21:11:46.601754 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:46.601643 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wns28\" (UniqueName: \"kubernetes.io/projected/50f16bde-5aa7-46f6-b801-68d84fa02ae3-kube-api-access-wns28\") pod \"payload-processing-e2e-api-a-3e92e3-85f886c5dd-c5pzb\" (UID: \"50f16bde-5aa7-46f6-b801-68d84fa02ae3\") " pod="openshift-ingress/payload-processing-e2e-api-a-3e92e3-85f886c5dd-c5pzb" Nov 03 21:11:46.602237 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:46.602205 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f16bde-5aa7-46f6-b801-68d84fa02ae3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3e92e3-85f886c5dd-c5pzb\" (UID: \"50f16bde-5aa7-46f6-b801-68d84fa02ae3\") " pod="openshift-ingress/payload-processing-e2e-api-a-3e92e3-85f886c5dd-c5pzb" Nov 03 21:11:46.619626 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:46.618169 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f16bde-5aa7-46f6-b801-68d84fa02ae3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3e92e3-85f886c5dd-c5pzb\" (UID: \"50f16bde-5aa7-46f6-b801-68d84fa02ae3\") " pod="openshift-ingress/payload-processing-e2e-api-a-3e92e3-85f886c5dd-c5pzb" Nov 03 21:11:46.742549 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:46.742502 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wns28\" (UniqueName: \"kubernetes.io/projected/50f16bde-5aa7-46f6-b801-68d84fa02ae3-kube-api-access-wns28\") pod \"payload-processing-e2e-api-a-3e92e3-85f886c5dd-c5pzb\" (UID: \"50f16bde-5aa7-46f6-b801-68d84fa02ae3\") " pod="openshift-ingress/payload-processing-e2e-api-a-3e92e3-85f886c5dd-c5pzb" Nov 03 21:11:46.805738 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:46.805710 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3e92e3-85f886c5dd-c5pzb" Nov 03 21:11:47.182853 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:47.182817 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3e92e3-85f886c5dd-c5pzb"] Nov 03 21:11:47.183248 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:11:47.183222 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f16bde_5aa7_46f6_b801_68d84fa02ae3.slice/crio-3a7b96065a6ae090bf7dbf2022961f7e2008f3a60798c86b326d5e93bcf6a39f WatchSource:0}: Error finding container 3a7b96065a6ae090bf7dbf2022961f7e2008f3a60798c86b326d5e93bcf6a39f: Status 404 returned error can't find the container with id 3a7b96065a6ae090bf7dbf2022961f7e2008f3a60798c86b326d5e93bcf6a39f Nov 03 21:11:48.004804 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:48.004758 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3e92e3-85f886c5dd-c5pzb" event={"ID":"50f16bde-5aa7-46f6-b801-68d84fa02ae3","Type":"ContainerStarted","Data":"669bbf139390438bd069da8c01f1c8437a8e0bcfc021d293f1b7f7cb07200765"} Nov 03 21:11:48.005257 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:48.004840 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3e92e3-85f886c5dd-c5pzb" event={"ID":"50f16bde-5aa7-46f6-b801-68d84fa02ae3","Type":"ContainerStarted","Data":"3a7b96065a6ae090bf7dbf2022961f7e2008f3a60798c86b326d5e93bcf6a39f"} Nov 03 21:11:48.005257 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:48.004871 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-3e92e3-85f886c5dd-c5pzb" Nov 03 21:11:48.116944 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:48.116877 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-3e92e3-85f886c5dd-c5pzb" podStartSLOduration=2.11685808 podStartE2EDuration="2.11685808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:11:48.114596406 +0000 UTC m=+1402.582375133" watchObservedRunningTime="2026-11-03 21:11:48.11685808 +0000 UTC m=+1402.584636805" Nov 03 21:11:51.301158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:51.301115 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3337df-6897659f76-nt5hq"] Nov 03 21:11:51.307207 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:51.307168 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3337df-6897659f76-nt5hq" Nov 03 21:11:51.365099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:51.364872 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3337df-6897659f76-nt5hq"] Nov 03 21:11:51.385101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:51.379273 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7hgf\" (UniqueName: \"kubernetes.io/projected/2e6932b4-504a-4fcf-8d86-0fda75a23e6b-kube-api-access-s7hgf\") pod \"payload-processing-e2e-api-b-3337df-6897659f76-nt5hq\" (UID: \"2e6932b4-504a-4fcf-8d86-0fda75a23e6b\") " pod="openshift-ingress/payload-processing-e2e-api-b-3337df-6897659f76-nt5hq" Nov 03 21:11:51.385101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:51.379497 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e6932b4-504a-4fcf-8d86-0fda75a23e6b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3337df-6897659f76-nt5hq\" (UID: \"2e6932b4-504a-4fcf-8d86-0fda75a23e6b\") " pod="openshift-ingress/payload-processing-e2e-api-b-3337df-6897659f76-nt5hq" Nov 03 21:11:51.570738 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:51.569339 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e6932b4-504a-4fcf-8d86-0fda75a23e6b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3337df-6897659f76-nt5hq\" (UID: \"2e6932b4-504a-4fcf-8d86-0fda75a23e6b\") " pod="openshift-ingress/payload-processing-e2e-api-b-3337df-6897659f76-nt5hq" Nov 03 21:11:51.572409 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:51.572362 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7hgf\" (UniqueName: \"kubernetes.io/projected/2e6932b4-504a-4fcf-8d86-0fda75a23e6b-kube-api-access-s7hgf\") pod \"payload-processing-e2e-api-b-3337df-6897659f76-nt5hq\" (UID: \"2e6932b4-504a-4fcf-8d86-0fda75a23e6b\") " pod="openshift-ingress/payload-processing-e2e-api-b-3337df-6897659f76-nt5hq" Nov 03 21:11:51.580719 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:51.577767 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-3337df-dockercfg-9j9gt\"" Nov 03 21:11:51.590770 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:51.581920 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-3337df\"" Nov 03 21:11:51.590770 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:51.587597 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e6932b4-504a-4fcf-8d86-0fda75a23e6b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3337df-6897659f76-nt5hq\" (UID: \"2e6932b4-504a-4fcf-8d86-0fda75a23e6b\") " pod="openshift-ingress/payload-processing-e2e-api-b-3337df-6897659f76-nt5hq" Nov 03 21:11:51.630811 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:51.630776 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7hgf\" (UniqueName: \"kubernetes.io/projected/2e6932b4-504a-4fcf-8d86-0fda75a23e6b-kube-api-access-s7hgf\") pod \"payload-processing-e2e-api-b-3337df-6897659f76-nt5hq\" (UID: \"2e6932b4-504a-4fcf-8d86-0fda75a23e6b\") " pod="openshift-ingress/payload-processing-e2e-api-b-3337df-6897659f76-nt5hq" Nov 03 21:11:51.634683 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:51.634654 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3337df-6897659f76-nt5hq" Nov 03 21:11:52.006209 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:52.006175 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3337df-6897659f76-nt5hq"] Nov 03 21:11:52.007686 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:11:52.007656 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e6932b4_504a_4fcf_8d86_0fda75a23e6b.slice/crio-5223e6894306bc59ce5665ed91164597be1a0f79e36d777f7391b3ccdf6ddc2c WatchSource:0}: Error finding container 5223e6894306bc59ce5665ed91164597be1a0f79e36d777f7391b3ccdf6ddc2c: Status 404 returned error can't find the container with id 5223e6894306bc59ce5665ed91164597be1a0f79e36d777f7391b3ccdf6ddc2c Nov 03 21:11:52.087085 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:52.087032 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3337df-6897659f76-nt5hq" event={"ID":"2e6932b4-504a-4fcf-8d86-0fda75a23e6b","Type":"ContainerStarted","Data":"5223e6894306bc59ce5665ed91164597be1a0f79e36d777f7391b3ccdf6ddc2c"} Nov 03 21:11:53.095368 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:53.095329 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3337df-6897659f76-nt5hq" event={"ID":"2e6932b4-504a-4fcf-8d86-0fda75a23e6b","Type":"ContainerStarted","Data":"95e04ad8f281e383a49d86ee8377e198e5e98e2c6707218547fe3f19da109e65"} Nov 03 21:11:53.095803 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:53.095530 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-3337df-6897659f76-nt5hq" Nov 03 21:11:53.207175 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:53.207120 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-3337df-6897659f76-nt5hq" podStartSLOduration=2.207107556 podStartE2EDuration="2.207107556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:11:53.205571727 +0000 UTC m=+1407.673350428" watchObservedRunningTime="2026-11-03 21:11:53.207107556 +0000 UTC m=+1407.674886278" Nov 03 21:11:59.015156 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:11:59.015118 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-3e92e3-85f886c5dd-c5pzb" Nov 03 21:12:04.106189 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:04.106156 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-3337df-6897659f76-nt5hq" Nov 03 21:12:10.620456 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:12:10.620415 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:12:31.312682 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:31.312632 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv"] Nov 03 21:12:31.318482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:31.318446 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv" Nov 03 21:12:31.389950 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:31.389901 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr8q4\" (UniqueName: \"kubernetes.io/projected/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-kube-api-access-wr8q4\") pod \"maas-api-e2e-api-b-3337df-54bd8f479d-497kv\" (UID: \"51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv" Nov 03 21:12:31.390185 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:31.390157 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-maas-api-tls\") pod \"maas-api-e2e-api-b-3337df-54bd8f479d-497kv\" (UID: \"51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv" Nov 03 21:12:31.390299 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:31.390280 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-metrics-certs\") pod \"maas-api-e2e-api-b-3337df-54bd8f479d-497kv\" (UID: \"51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv" Nov 03 21:12:31.390380 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:31.390353 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-openshift-service-ca\") pod \"maas-api-e2e-api-b-3337df-54bd8f479d-497kv\" (UID: \"51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv" Nov 03 21:12:31.415721 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:31.415681 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-3337df-serving-cert\"" Nov 03 21:12:31.419332 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:31.419302 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv"] Nov 03 21:12:31.492429 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:31.492394 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-metrics-certs\") pod \"maas-api-e2e-api-b-3337df-54bd8f479d-497kv\" (UID: \"51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv" Nov 03 21:12:31.492429 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:31.492436 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-openshift-service-ca\") pod \"maas-api-e2e-api-b-3337df-54bd8f479d-497kv\" (UID: \"51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv" Nov 03 21:12:31.492725 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:31.492605 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr8q4\" (UniqueName: \"kubernetes.io/projected/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-kube-api-access-wr8q4\") pod \"maas-api-e2e-api-b-3337df-54bd8f479d-497kv\" (UID: \"51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv" Nov 03 21:12:31.492793 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:31.492723 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-maas-api-tls\") pod \"maas-api-e2e-api-b-3337df-54bd8f479d-497kv\" (UID: \"51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv" Nov 03 21:12:31.492997 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:12:31.492969 5615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-serving-cert: secret "maas-api-e2e-api-b-3337df-serving-cert" not found Nov 03 21:12:31.493150 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:12:31.493044 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-maas-api-tls podName:51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91 nodeName:}" failed. No retries permitted until 2026-11-03 21:12:31.993025465 +0000 UTC m=+1446.460804183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-maas-api-tls") pod "maas-api-e2e-api-b-3337df-54bd8f479d-497kv" (UID: "51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91") : secret "maas-api-e2e-api-b-3337df-serving-cert" not found Nov 03 21:12:31.493400 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:31.493355 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-openshift-service-ca\") pod \"maas-api-e2e-api-b-3337df-54bd8f479d-497kv\" (UID: \"51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv" Nov 03 21:12:31.504355 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:31.504325 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-metrics-certs\") pod \"maas-api-e2e-api-b-3337df-54bd8f479d-497kv\" (UID: \"51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv" Nov 03 21:12:31.553434 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:31.553395 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr8q4\" (UniqueName: \"kubernetes.io/projected/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-kube-api-access-wr8q4\") pod \"maas-api-e2e-api-b-3337df-54bd8f479d-497kv\" (UID: \"51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv" Nov 03 21:12:32.002315 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:32.002273 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-maas-api-tls\") pod \"maas-api-e2e-api-b-3337df-54bd8f479d-497kv\" (UID: \"51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv" Nov 03 21:12:32.005096 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:32.005046 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-maas-api-tls\") pod \"maas-api-e2e-api-b-3337df-54bd8f479d-497kv\" (UID: \"51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv" Nov 03 21:12:32.236114 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:32.236066 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv" Nov 03 21:12:32.451376 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:32.451293 5615 generic.go:356] "Generic (PLEG): container finished" podID="a388fecb-f964-46d3-a07e-4a87bf4c8981" containerID="1cfb6ca4f7f451d6f08c6e8d975894c03497943625786f17f4c97add819261b8" exitCode=0 Nov 03 21:12:32.451757 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:32.451370 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" event={"ID":"a388fecb-f964-46d3-a07e-4a87bf4c8981","Type":"ContainerDied","Data":"1cfb6ca4f7f451d6f08c6e8d975894c03497943625786f17f4c97add819261b8"} Nov 03 21:12:32.451974 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:32.451805 5615 scope.go:122] "RemoveContainer" containerID="1cfb6ca4f7f451d6f08c6e8d975894c03497943625786f17f4c97add819261b8" Nov 03 21:12:32.453410 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:32.453382 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:12:32.453475 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:32.453448 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:12:32.453512 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:32.453482 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:12:32.666974 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:32.665908 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv"] Nov 03 21:12:33.538099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:33.532648 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv" event={"ID":"51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91","Type":"ContainerStarted","Data":"d775f4ac7fe4b0871e57117e116015830382f59be486d51974f61eaf4bccdb8b"} Nov 03 21:12:33.542532 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:33.542500 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" event={"ID":"a388fecb-f964-46d3-a07e-4a87bf4c8981","Type":"ContainerStarted","Data":"fe3c5404b12c1f35774178afcfc12cb47a373e22f4311a68e145fc8f98e171b7"} Nov 03 21:12:34.025762 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:34.025699 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:12:34.026856 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:34.026823 5615 patch_prober.go:29] interesting pod/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 21:12:34.027014 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:34.026884 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" podUID="a388fecb-f964-46d3-a07e-4a87bf4c8981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 21:12:34.550680 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:34.550641 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv" event={"ID":"51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91","Type":"ContainerStarted","Data":"aa65af3220b554e1e7fef2e8e8f2a7980f29280f301546d4e5c7b937f211c644"} Nov 03 21:12:34.551154 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:34.550740 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv" Nov 03 21:12:34.683607 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:34.683539 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv" podStartSLOduration=2.721617914 podStartE2EDuration="3.683525112s" totalImagesPullingTime="961.907198ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:12:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:12:34.68120927 +0000 UTC m=+1449.148987996" watchObservedRunningTime="2026-11-03 21:12:34.683525112 +0000 UTC m=+1449.151303834" Nov 03 21:12:35.026431 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:35.026393 5615 patch_prober.go:29] interesting pod/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 21:12:35.026649 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:35.026452 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" podUID="a388fecb-f964-46d3-a07e-4a87bf4c8981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 21:12:36.026363 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:36.025711 5615 patch_prober.go:29] interesting pod/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 21:12:36.026363 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:36.025771 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" podUID="a388fecb-f964-46d3-a07e-4a87bf4c8981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 21:12:37.025826 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:37.025788 5615 patch_prober.go:29] interesting pod/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 21:12:37.026064 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:37.025852 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" podUID="a388fecb-f964-46d3-a07e-4a87bf4c8981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 21:12:37.717243 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:37.717195 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:12:38.036099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:38.035473 5615 patch_prober.go:29] interesting pod/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 21:12:38.036099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:38.035536 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" podUID="a388fecb-f964-46d3-a07e-4a87bf4c8981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 21:12:39.025148 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:39.025105 5615 patch_prober.go:29] interesting pod/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 21:12:39.025582 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:39.025186 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" podUID="a388fecb-f964-46d3-a07e-4a87bf4c8981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 21:12:40.031089 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:40.031044 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:12:40.032188 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:40.032165 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:12:40.564301 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:40.564267 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv" Nov 03 21:12:40.647056 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:12:40.647023 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:12:49.693528 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:49.693492 5615 generic.go:356] "Generic (PLEG): container finished" podID="5ba344bf-b7b1-4bfe-8098-78073ecc5fb6" containerID="499d514c2ee1c87367e85a5b68ecf0f9c459b88c03fc834bc7b1265a8cd3ad79" exitCode=0 Nov 03 21:12:49.693970 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:49.693570 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" event={"ID":"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6","Type":"ContainerDied","Data":"499d514c2ee1c87367e85a5b68ecf0f9c459b88c03fc834bc7b1265a8cd3ad79"} Nov 03 21:12:49.693970 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:49.693966 5615 scope.go:122] "RemoveContainer" containerID="499d514c2ee1c87367e85a5b68ecf0f9c459b88c03fc834bc7b1265a8cd3ad79" Nov 03 21:12:49.695461 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:49.695428 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:12:49.695567 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:49.695502 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:12:49.695567 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:49.695533 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:12:50.728260 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:50.728218 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" event={"ID":"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6","Type":"ContainerStarted","Data":"64907aa41b720db14fffe3e1b295a757c90b094c329d04a21a3f025096ed5f9d"} Nov 03 21:12:50.952288 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:50.952244 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:12:50.953782 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:50.953759 5615 patch_prober.go:29] interesting pod/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 21:12:50.953857 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:50.953823 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" podUID="5ba344bf-b7b1-4bfe-8098-78073ecc5fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 21:12:51.855741 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:51.855706 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:12:51.952992 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:51.952953 5615 patch_prober.go:29] interesting pod/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 21:12:51.953197 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:51.953008 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" podUID="5ba344bf-b7b1-4bfe-8098-78073ecc5fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 21:12:52.770749 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:52.770709 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-kj8mx"] Nov 03 21:12:52.952575 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:52.952536 5615 patch_prober.go:29] interesting pod/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 21:12:52.952968 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:52.952600 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" podUID="5ba344bf-b7b1-4bfe-8098-78073ecc5fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 21:12:53.953040 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:53.952999 5615 patch_prober.go:29] interesting pod/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 21:12:53.953482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:53.953058 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" podUID="5ba344bf-b7b1-4bfe-8098-78073ecc5fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 21:12:54.953101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:54.953034 5615 patch_prober.go:29] interesting pod/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 21:12:54.953522 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:54.953105 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" podUID="5ba344bf-b7b1-4bfe-8098-78073ecc5fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 21:12:56.053826 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:56.053785 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:12:56.055979 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:56.055949 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:12:57.771213 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:57.771143 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-kj8mx" podUID="afb96de0-be6d-452d-bb92-5b72b3563cb6" containerName="payload-processing" containerID="cri-o://29d81ce462c869f58b307512a93c554ee348e65a8a11f7faaeead64c09ac808c" gracePeriod=25 Nov 03 21:12:58.046709 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:58.046681 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-kj8mx" Nov 03 21:12:58.108417 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:58.108384 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afb96de0-be6d-452d-bb92-5b72b3563cb6-kube-api-access-2bwlm\" (UniqueName: \"kubernetes.io/projected/afb96de0-be6d-452d-bb92-5b72b3563cb6-kube-api-access-2bwlm\") pod \"afb96de0-be6d-452d-bb92-5b72b3563cb6\" (UID: \"afb96de0-be6d-452d-bb92-5b72b3563cb6\") " Nov 03 21:12:58.108714 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:58.108691 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afb96de0-be6d-452d-bb92-5b72b3563cb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afb96de0-be6d-452d-bb92-5b72b3563cb6-plugins-config-volume\") pod \"afb96de0-be6d-452d-bb92-5b72b3563cb6\" (UID: \"afb96de0-be6d-452d-bb92-5b72b3563cb6\") " Nov 03 21:12:58.109002 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:58.108976 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afb96de0-be6d-452d-bb92-5b72b3563cb6-plugins-config-volume" pod "afb96de0-be6d-452d-bb92-5b72b3563cb6" (UID: "afb96de0-be6d-452d-bb92-5b72b3563cb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:12:58.109302 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:58.109285 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afb96de0-be6d-452d-bb92-5b72b3563cb6-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:58.110588 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:58.110560 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb96de0-be6d-452d-bb92-5b72b3563cb6-kube-api-access-2bwlm" pod "afb96de0-be6d-452d-bb92-5b72b3563cb6" (UID: "afb96de0-be6d-452d-bb92-5b72b3563cb6"). InnerVolumeSpecName "kube-api-access-2bwlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:12:58.210875 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:58.210835 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bwlm\" (UniqueName: \"kubernetes.io/projected/afb96de0-be6d-452d-bb92-5b72b3563cb6-kube-api-access-2bwlm\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:58.805572 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:58.805526 5615 generic.go:356] "Generic (PLEG): container finished" podID="afb96de0-be6d-452d-bb92-5b72b3563cb6" containerID="29d81ce462c869f58b307512a93c554ee348e65a8a11f7faaeead64c09ac808c" exitCode=0 Nov 03 21:12:58.806055 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:58.805602 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-kj8mx" Nov 03 21:12:58.806055 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:58.805608 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-kj8mx" event={"ID":"afb96de0-be6d-452d-bb92-5b72b3563cb6","Type":"ContainerDied","Data":"29d81ce462c869f58b307512a93c554ee348e65a8a11f7faaeead64c09ac808c"} Nov 03 21:12:58.806055 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:58.805666 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-kj8mx" event={"ID":"afb96de0-be6d-452d-bb92-5b72b3563cb6","Type":"ContainerDied","Data":"e39496f8ec34ed6da1098429975d224de9fbc09a756621aa5419e2d5fa7a3618"} Nov 03 21:12:58.806055 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:58.805689 5615 scope.go:122] "RemoveContainer" containerID="29d81ce462c869f58b307512a93c554ee348e65a8a11f7faaeead64c09ac808c" Nov 03 21:12:58.816560 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:58.816540 5615 scope.go:122] "RemoveContainer" containerID="29d81ce462c869f58b307512a93c554ee348e65a8a11f7faaeead64c09ac808c" Nov 03 21:12:58.816855 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:12:58.816832 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29d81ce462c869f58b307512a93c554ee348e65a8a11f7faaeead64c09ac808c\": container with ID starting with 29d81ce462c869f58b307512a93c554ee348e65a8a11f7faaeead64c09ac808c not found: ID does not exist" containerID="29d81ce462c869f58b307512a93c554ee348e65a8a11f7faaeead64c09ac808c" Nov 03 21:12:58.816902 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:58.816864 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29d81ce462c869f58b307512a93c554ee348e65a8a11f7faaeead64c09ac808c"} err="failed to get container status \"29d81ce462c869f58b307512a93c554ee348e65a8a11f7faaeead64c09ac808c\": rpc error: code = NotFound desc = could not find container \"29d81ce462c869f58b307512a93c554ee348e65a8a11f7faaeead64c09ac808c\": container with ID starting with 29d81ce462c869f58b307512a93c554ee348e65a8a11f7faaeead64c09ac808c not found: ID does not exist" Nov 03 21:12:58.962879 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:58.962849 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-kj8mx"] Nov 03 21:12:58.966001 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:58.965976 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-kj8mx"] Nov 03 21:12:59.696096 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:12:59.684500 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afb96de0-be6d-452d-bb92-5b72b3563cb6" path="/var/lib/kubelet/pods/afb96de0-be6d-452d-bb92-5b72b3563cb6/volumes" Nov 03 21:13:10.619854 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:13:10.619806 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:13:39.751099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:39.750817 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-22600c-68c974ccf4-kp5mr"] Nov 03 21:13:39.758918 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:39.751949 5615 cpu_manager.go:414] "containerMap: removing container" podUID="afb96de0-be6d-452d-bb92-5b72b3563cb6" containerName="payload-processing" Nov 03 21:13:39.759226 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:39.759168 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afb96de0-be6d-452d-bb92-5b72b3563cb6" containerName="payload-processing" Nov 03 21:13:39.760677 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:39.760654 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="afb96de0-be6d-452d-bb92-5b72b3563cb6" containerName="payload-processing" Nov 03 21:13:39.774245 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:39.774211 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-22600c-68c974ccf4-kp5mr"] Nov 03 21:13:39.774637 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:39.774615 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-22600c-68c974ccf4-kp5mr" Nov 03 21:13:39.916254 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:39.910286 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-22600c\"" Nov 03 21:13:39.928053 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:39.927970 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz7fj\" (UniqueName: \"kubernetes.io/projected/2858cbf3-8624-4362-8997-e62ae6013b53-kube-api-access-jz7fj\") pod \"payload-processing-e2e-api-b-22600c-68c974ccf4-kp5mr\" (UID: \"2858cbf3-8624-4362-8997-e62ae6013b53\") " pod="openshift-ingress/payload-processing-e2e-api-b-22600c-68c974ccf4-kp5mr" Nov 03 21:13:39.928432 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:39.928408 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2858cbf3-8624-4362-8997-e62ae6013b53-plugins-config-volume\") pod \"payload-processing-e2e-api-b-22600c-68c974ccf4-kp5mr\" (UID: \"2858cbf3-8624-4362-8997-e62ae6013b53\") " pod="openshift-ingress/payload-processing-e2e-api-b-22600c-68c974ccf4-kp5mr" Nov 03 21:13:39.948173 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:39.947478 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-22600c-dockercfg-szgg5\"" Nov 03 21:13:40.030145 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:40.030038 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz7fj\" (UniqueName: \"kubernetes.io/projected/2858cbf3-8624-4362-8997-e62ae6013b53-kube-api-access-jz7fj\") pod \"payload-processing-e2e-api-b-22600c-68c974ccf4-kp5mr\" (UID: \"2858cbf3-8624-4362-8997-e62ae6013b53\") " pod="openshift-ingress/payload-processing-e2e-api-b-22600c-68c974ccf4-kp5mr" Nov 03 21:13:40.030324 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:40.030184 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2858cbf3-8624-4362-8997-e62ae6013b53-plugins-config-volume\") pod \"payload-processing-e2e-api-b-22600c-68c974ccf4-kp5mr\" (UID: \"2858cbf3-8624-4362-8997-e62ae6013b53\") " pod="openshift-ingress/payload-processing-e2e-api-b-22600c-68c974ccf4-kp5mr" Nov 03 21:13:40.030951 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:40.030921 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2858cbf3-8624-4362-8997-e62ae6013b53-plugins-config-volume\") pod \"payload-processing-e2e-api-b-22600c-68c974ccf4-kp5mr\" (UID: \"2858cbf3-8624-4362-8997-e62ae6013b53\") " pod="openshift-ingress/payload-processing-e2e-api-b-22600c-68c974ccf4-kp5mr" Nov 03 21:13:40.088378 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:40.088341 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz7fj\" (UniqueName: \"kubernetes.io/projected/2858cbf3-8624-4362-8997-e62ae6013b53-kube-api-access-jz7fj\") pod \"payload-processing-e2e-api-b-22600c-68c974ccf4-kp5mr\" (UID: \"2858cbf3-8624-4362-8997-e62ae6013b53\") " pod="openshift-ingress/payload-processing-e2e-api-b-22600c-68c974ccf4-kp5mr" Nov 03 21:13:40.251736 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:40.251692 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-22600c-68c974ccf4-kp5mr" Nov 03 21:13:40.619951 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:13:40.619869 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:13:40.639032 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:40.639006 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-22600c-68c974ccf4-kp5mr"] Nov 03 21:13:40.640566 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:13:40.640530 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2858cbf3_8624_4362_8997_e62ae6013b53.slice/crio-2c7243ebb2bbf84dff6b4187948763e414b2dc1f8639f4db8e85d260bdab0aca WatchSource:0}: Error finding container 2c7243ebb2bbf84dff6b4187948763e414b2dc1f8639f4db8e85d260bdab0aca: Status 404 returned error can't find the container with id 2c7243ebb2bbf84dff6b4187948763e414b2dc1f8639f4db8e85d260bdab0aca Nov 03 21:13:41.574141 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:41.574059 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-22600c-68c974ccf4-kp5mr" event={"ID":"2858cbf3-8624-4362-8997-e62ae6013b53","Type":"ContainerStarted","Data":"43d93e4a7e8fa171047b1062c3f372e2b5d0923dfaa14533727f4ff82d906409"} Nov 03 21:13:41.574632 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:41.574152 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-22600c-68c974ccf4-kp5mr" event={"ID":"2858cbf3-8624-4362-8997-e62ae6013b53","Type":"ContainerStarted","Data":"2c7243ebb2bbf84dff6b4187948763e414b2dc1f8639f4db8e85d260bdab0aca"} Nov 03 21:13:41.574632 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:41.574180 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-22600c-68c974ccf4-kp5mr" Nov 03 21:13:41.730835 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:41.730768 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-22600c-68c974ccf4-kp5mr" podStartSLOduration=2.730747657 podStartE2EDuration="2.730747657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:13:41.726461649 +0000 UTC m=+1516.194240372" watchObservedRunningTime="2026-11-03 21:13:41.730747657 +0000 UTC m=+1516.198526382" Nov 03 21:13:43.058143 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:43.058099 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22"] Nov 03 21:13:43.058679 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:43.058378 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" podUID="02a77aa8-85a7-434b-9724-4e787da56d34" containerName="istio-proxy" containerID="cri-o://686332be9dc4bd010a266ec71e65074d5f7b4978f4fc8ec7697f7b105fadb006" gracePeriod=30 Nov 03 21:13:44.478654 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:44.478609 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6dacc6-78c86d8fb4-9hmzf"] Nov 03 21:13:44.588897 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:44.588849 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6dacc6-5d69b9ffff-dwd78"] Nov 03 21:13:48.359626 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.359596 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:13:48.489765 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.489714 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-istio-data\") pod \"02a77aa8-85a7-434b-9724-4e787da56d34\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " Nov 03 21:13:48.489947 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.489794 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02a77aa8-85a7-434b-9724-4e787da56d34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02a77aa8-85a7-434b-9724-4e787da56d34-istiod-ca-cert\") pod \"02a77aa8-85a7-434b-9724-4e787da56d34\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " Nov 03 21:13:48.489947 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.489873 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-workload-certs\") pod \"02a77aa8-85a7-434b-9724-4e787da56d34\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " Nov 03 21:13:48.489947 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.489905 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-credential-socket\") pod \"02a77aa8-85a7-434b-9724-4e787da56d34\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " Nov 03 21:13:48.490147 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.489964 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02a77aa8-85a7-434b-9724-4e787da56d34-kube-api-access-lktd5\" (UniqueName: \"kubernetes.io/projected/02a77aa8-85a7-434b-9724-4e787da56d34-kube-api-access-lktd5\") pod \"02a77aa8-85a7-434b-9724-4e787da56d34\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " Nov 03 21:13:48.490147 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.490034 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-istio-data" pod "02a77aa8-85a7-434b-9724-4e787da56d34" (UID: "02a77aa8-85a7-434b-9724-4e787da56d34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:48.490147 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.490063 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-istio-envoy\") pod \"02a77aa8-85a7-434b-9724-4e787da56d34\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " Nov 03 21:13:48.490265 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.490189 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-workload-socket\") pod \"02a77aa8-85a7-434b-9724-4e787da56d34\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " Nov 03 21:13:48.490298 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.490265 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-workload-certs" pod "02a77aa8-85a7-434b-9724-4e787da56d34" (UID: "02a77aa8-85a7-434b-9724-4e787da56d34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:48.490995 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.490352 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02a77aa8-85a7-434b-9724-4e787da56d34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02a77aa8-85a7-434b-9724-4e787da56d34-istio-podinfo\") pod \"02a77aa8-85a7-434b-9724-4e787da56d34\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " Nov 03 21:13:48.490995 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.490405 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02a77aa8-85a7-434b-9724-4e787da56d34-istio-token\" (UniqueName: \"kubernetes.io/projected/02a77aa8-85a7-434b-9724-4e787da56d34-istio-token\") pod \"02a77aa8-85a7-434b-9724-4e787da56d34\" (UID: \"02a77aa8-85a7-434b-9724-4e787da56d34\") " Nov 03 21:13:48.490995 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.490606 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02a77aa8-85a7-434b-9724-4e787da56d34-istiod-ca-cert" pod "02a77aa8-85a7-434b-9724-4e787da56d34" (UID: "02a77aa8-85a7-434b-9724-4e787da56d34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:48.490995 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.490604 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-credential-socket" pod "02a77aa8-85a7-434b-9724-4e787da56d34" (UID: "02a77aa8-85a7-434b-9724-4e787da56d34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:48.490995 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.490873 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-workload-socket" pod "02a77aa8-85a7-434b-9724-4e787da56d34" (UID: "02a77aa8-85a7-434b-9724-4e787da56d34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:48.491753 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.491655 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:48.491753 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.491684 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02a77aa8-85a7-434b-9724-4e787da56d34-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:48.491753 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.491700 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:48.491753 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.491714 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:48.491753 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.491728 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:48.494494 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.494456 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02a77aa8-85a7-434b-9724-4e787da56d34-istio-podinfo" pod "02a77aa8-85a7-434b-9724-4e787da56d34" (UID: "02a77aa8-85a7-434b-9724-4e787da56d34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:13:48.494494 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.494474 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02a77aa8-85a7-434b-9724-4e787da56d34-kube-api-access-lktd5" pod "02a77aa8-85a7-434b-9724-4e787da56d34" (UID: "02a77aa8-85a7-434b-9724-4e787da56d34"). InnerVolumeSpecName "kube-api-access-lktd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:48.495451 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.495425 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02a77aa8-85a7-434b-9724-4e787da56d34-istio-token" pod "02a77aa8-85a7-434b-9724-4e787da56d34" (UID: "02a77aa8-85a7-434b-9724-4e787da56d34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:48.495560 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.495539 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-istio-envoy" pod "02a77aa8-85a7-434b-9724-4e787da56d34" (UID: "02a77aa8-85a7-434b-9724-4e787da56d34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:48.651586 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.651493 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lktd5\" (UniqueName: \"kubernetes.io/projected/02a77aa8-85a7-434b-9724-4e787da56d34-kube-api-access-lktd5\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:48.651819 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.651796 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02a77aa8-85a7-434b-9724-4e787da56d34-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:48.651932 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.651917 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02a77aa8-85a7-434b-9724-4e787da56d34-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:48.652050 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.652035 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02a77aa8-85a7-434b-9724-4e787da56d34-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:48.699155 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.695712 5615 generic.go:356] "Generic (PLEG): container finished" podID="02a77aa8-85a7-434b-9724-4e787da56d34" containerID="686332be9dc4bd010a266ec71e65074d5f7b4978f4fc8ec7697f7b105fadb006" exitCode=0 Nov 03 21:13:48.699155 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.695879 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" event={"ID":"02a77aa8-85a7-434b-9724-4e787da56d34","Type":"ContainerDied","Data":"686332be9dc4bd010a266ec71e65074d5f7b4978f4fc8ec7697f7b105fadb006"} Nov 03 21:13:48.699155 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.695945 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" event={"ID":"02a77aa8-85a7-434b-9724-4e787da56d34","Type":"ContainerDied","Data":"25e647886071e31ef63f5264086462a7014433f42b4b28908a52551b873b2173"} Nov 03 21:13:48.699155 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.695977 5615 scope.go:122] "RemoveContainer" containerID="686332be9dc4bd010a266ec71e65074d5f7b4978f4fc8ec7697f7b105fadb006" Nov 03 21:13:48.699155 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.696182 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22" Nov 03 21:13:48.746532 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.746481 5615 scope.go:122] "RemoveContainer" containerID="146e57fd4b068f61ad45c53e0244223188aa84574882580d5240c4699b44f9d7" Nov 03 21:13:48.763308 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.763281 5615 scope.go:122] "RemoveContainer" containerID="686332be9dc4bd010a266ec71e65074d5f7b4978f4fc8ec7697f7b105fadb006" Nov 03 21:13:48.763676 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:13:48.763650 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"686332be9dc4bd010a266ec71e65074d5f7b4978f4fc8ec7697f7b105fadb006\": container with ID starting with 686332be9dc4bd010a266ec71e65074d5f7b4978f4fc8ec7697f7b105fadb006 not found: ID does not exist" containerID="686332be9dc4bd010a266ec71e65074d5f7b4978f4fc8ec7697f7b105fadb006" Nov 03 21:13:48.763769 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.763697 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686332be9dc4bd010a266ec71e65074d5f7b4978f4fc8ec7697f7b105fadb006"} err="failed to get container status \"686332be9dc4bd010a266ec71e65074d5f7b4978f4fc8ec7697f7b105fadb006\": rpc error: code = NotFound desc = could not find container \"686332be9dc4bd010a266ec71e65074d5f7b4978f4fc8ec7697f7b105fadb006\": container with ID starting with 686332be9dc4bd010a266ec71e65074d5f7b4978f4fc8ec7697f7b105fadb006 not found: ID does not exist" Nov 03 21:13:48.763769 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.763719 5615 scope.go:122] "RemoveContainer" containerID="146e57fd4b068f61ad45c53e0244223188aa84574882580d5240c4699b44f9d7" Nov 03 21:13:48.764554 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:13:48.764068 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"146e57fd4b068f61ad45c53e0244223188aa84574882580d5240c4699b44f9d7\": container with ID starting with 146e57fd4b068f61ad45c53e0244223188aa84574882580d5240c4699b44f9d7 not found: ID does not exist" containerID="146e57fd4b068f61ad45c53e0244223188aa84574882580d5240c4699b44f9d7" Nov 03 21:13:48.764706 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.764678 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"146e57fd4b068f61ad45c53e0244223188aa84574882580d5240c4699b44f9d7"} err="failed to get container status \"146e57fd4b068f61ad45c53e0244223188aa84574882580d5240c4699b44f9d7\": rpc error: code = NotFound desc = could not find container \"146e57fd4b068f61ad45c53e0244223188aa84574882580d5240c4699b44f9d7\": container with ID starting with 146e57fd4b068f61ad45c53e0244223188aa84574882580d5240c4699b44f9d7 not found: ID does not exist" Nov 03 21:13:48.847645 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.847609 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-69f498847-h225h"] Nov 03 21:13:48.848386 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.848368 5615 cpu_manager.go:414] "containerMap: removing container" podUID="02a77aa8-85a7-434b-9724-4e787da56d34" containerName="istio-proxy" Nov 03 21:13:48.848438 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.848389 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02a77aa8-85a7-434b-9724-4e787da56d34" containerName="istio-proxy" Nov 03 21:13:48.848438 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.848401 5615 cpu_manager.go:414] "containerMap: removing container" podUID="02a77aa8-85a7-434b-9724-4e787da56d34" containerName="istio-proxy" Nov 03 21:13:48.848438 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.848406 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02a77aa8-85a7-434b-9724-4e787da56d34" containerName="istio-proxy" Nov 03 21:13:48.848564 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.848554 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="02a77aa8-85a7-434b-9724-4e787da56d34" containerName="istio-proxy" Nov 03 21:13:48.848618 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.848566 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="02a77aa8-85a7-434b-9724-4e787da56d34" containerName="istio-proxy" Nov 03 21:13:48.852643 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.852625 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-69f498847-h225h" Nov 03 21:13:48.864884 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.864853 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22"] Nov 03 21:13:48.870976 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.870948 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-145cc5-openshift-default-d67446f7f-dxt22"] Nov 03 21:13:48.902778 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.902689 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0f8726-dockercfg-5tlxk\"" Nov 03 21:13:48.902778 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.902754 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0f8726\"" Nov 03 21:13:48.905568 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:48.905530 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-69f498847-h225h"] Nov 03 21:13:49.076097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:49.068065 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce818536-2d2c-4aa1-865d-1921957f995e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0f8726-69f498847-h225h\" (UID: \"ce818536-2d2c-4aa1-865d-1921957f995e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-69f498847-h225h" Nov 03 21:13:49.076097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:49.068415 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8bpf\" (UniqueName: \"kubernetes.io/projected/ce818536-2d2c-4aa1-865d-1921957f995e-kube-api-access-w8bpf\") pod \"payload-pre-processing-e2e-ipp-a-0f8726-69f498847-h225h\" (UID: \"ce818536-2d2c-4aa1-865d-1921957f995e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-69f498847-h225h" Nov 03 21:13:49.229153 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:49.222985 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce818536-2d2c-4aa1-865d-1921957f995e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0f8726-69f498847-h225h\" (UID: \"ce818536-2d2c-4aa1-865d-1921957f995e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-69f498847-h225h" Nov 03 21:13:49.229153 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:49.223334 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8bpf\" (UniqueName: \"kubernetes.io/projected/ce818536-2d2c-4aa1-865d-1921957f995e-kube-api-access-w8bpf\") pod \"payload-pre-processing-e2e-ipp-a-0f8726-69f498847-h225h\" (UID: \"ce818536-2d2c-4aa1-865d-1921957f995e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-69f498847-h225h" Nov 03 21:13:49.229153 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:49.224606 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce818536-2d2c-4aa1-865d-1921957f995e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0f8726-69f498847-h225h\" (UID: \"ce818536-2d2c-4aa1-865d-1921957f995e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-69f498847-h225h" Nov 03 21:13:49.257102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:49.248246 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0f8726-6878775966-spjk6"] Nov 03 21:13:49.271399 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:49.259247 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0f8726-6878775966-spjk6"] Nov 03 21:13:49.271399 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:49.260668 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-6878775966-spjk6" Nov 03 21:13:49.283692 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:49.283658 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8bpf\" (UniqueName: \"kubernetes.io/projected/ce818536-2d2c-4aa1-865d-1921957f995e-kube-api-access-w8bpf\") pod \"payload-pre-processing-e2e-ipp-a-0f8726-69f498847-h225h\" (UID: \"ce818536-2d2c-4aa1-865d-1921957f995e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-69f498847-h225h" Nov 03 21:13:49.324811 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:49.324772 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trm8x\" (UniqueName: \"kubernetes.io/projected/e0b123ee-6eab-443c-9a71-af860f6f5ace-kube-api-access-trm8x\") pod \"payload-processing-e2e-ipp-a-0f8726-6878775966-spjk6\" (UID: \"e0b123ee-6eab-443c-9a71-af860f6f5ace\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-6878775966-spjk6" Nov 03 21:13:49.325063 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:49.325038 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0b123ee-6eab-443c-9a71-af860f6f5ace-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0f8726-6878775966-spjk6\" (UID: \"e0b123ee-6eab-443c-9a71-af860f6f5ace\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-6878775966-spjk6" Nov 03 21:13:49.426711 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:49.426674 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trm8x\" (UniqueName: \"kubernetes.io/projected/e0b123ee-6eab-443c-9a71-af860f6f5ace-kube-api-access-trm8x\") pod \"payload-processing-e2e-ipp-a-0f8726-6878775966-spjk6\" (UID: \"e0b123ee-6eab-443c-9a71-af860f6f5ace\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-6878775966-spjk6" Nov 03 21:13:49.427147 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:49.426856 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0b123ee-6eab-443c-9a71-af860f6f5ace-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0f8726-6878775966-spjk6\" (UID: \"e0b123ee-6eab-443c-9a71-af860f6f5ace\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-6878775966-spjk6" Nov 03 21:13:49.427604 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:49.427580 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0b123ee-6eab-443c-9a71-af860f6f5ace-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0f8726-6878775966-spjk6\" (UID: \"e0b123ee-6eab-443c-9a71-af860f6f5ace\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-6878775966-spjk6" Nov 03 21:13:49.466808 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:49.466775 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-69f498847-h225h" Nov 03 21:13:49.479191 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:49.479123 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-6dacc6-78c86d8fb4-9hmzf" podUID="ff023bfe-ed10-4ea6-bef1-bb220444bf00" containerName="payload-processing" containerID="cri-o://57800c9abf3279ab463a195ade2d8c9a05d256bf77ac58cee3f10ae7d42951f7" gracePeriod=25 Nov 03 21:13:49.485305 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:49.485272 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trm8x\" (UniqueName: \"kubernetes.io/projected/e0b123ee-6eab-443c-9a71-af860f6f5ace-kube-api-access-trm8x\") pod \"payload-processing-e2e-ipp-a-0f8726-6878775966-spjk6\" (UID: \"e0b123ee-6eab-443c-9a71-af860f6f5ace\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-6878775966-spjk6" Nov 03 21:13:49.583881 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:49.583842 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-6878775966-spjk6" Nov 03 21:13:49.590132 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:49.589482 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dacc6-5d69b9ffff-dwd78" podUID="af9e936f-0ebb-449b-b00b-0716f2525039" containerName="payload-pre-processing" containerID="cri-o://bbe7c825d14d906ac41669677b5b687b3c332e08ff85f0001ed987db8baff451" gracePeriod=25 Nov 03 21:13:49.658770 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:49.658727 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02a77aa8-85a7-434b-9724-4e787da56d34" path="/var/lib/kubelet/pods/02a77aa8-85a7-434b-9724-4e787da56d34/volumes" Nov 03 21:13:49.711372 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:49.711342 5615 generic.go:356] "Generic (PLEG): container finished" podID="ff023bfe-ed10-4ea6-bef1-bb220444bf00" containerID="57800c9abf3279ab463a195ade2d8c9a05d256bf77ac58cee3f10ae7d42951f7" exitCode=0 Nov 03 21:13:49.711513 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:49.711488 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6dacc6-78c86d8fb4-9hmzf" event={"ID":"ff023bfe-ed10-4ea6-bef1-bb220444bf00","Type":"ContainerDied","Data":"57800c9abf3279ab463a195ade2d8c9a05d256bf77ac58cee3f10ae7d42951f7"} Nov 03 21:13:49.861040 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:49.855683 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6dacc6-78c86d8fb4-9hmzf" Nov 03 21:13:49.861040 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:49.857671 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-69f498847-h225h"] Nov 03 21:13:50.067981 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.060912 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff023bfe-ed10-4ea6-bef1-bb220444bf00-kube-api-access-5mfhh\" (UniqueName: \"kubernetes.io/projected/ff023bfe-ed10-4ea6-bef1-bb220444bf00-kube-api-access-5mfhh\") pod \"ff023bfe-ed10-4ea6-bef1-bb220444bf00\" (UID: \"ff023bfe-ed10-4ea6-bef1-bb220444bf00\") " Nov 03 21:13:50.067981 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.061224 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff023bfe-ed10-4ea6-bef1-bb220444bf00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff023bfe-ed10-4ea6-bef1-bb220444bf00-plugins-config-volume\") pod \"ff023bfe-ed10-4ea6-bef1-bb220444bf00\" (UID: \"ff023bfe-ed10-4ea6-bef1-bb220444bf00\") " Nov 03 21:13:50.067981 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.061924 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0f8726-6878775966-spjk6"] Nov 03 21:13:50.067981 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.062310 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff023bfe-ed10-4ea6-bef1-bb220444bf00-plugins-config-volume" pod "ff023bfe-ed10-4ea6-bef1-bb220444bf00" (UID: "ff023bfe-ed10-4ea6-bef1-bb220444bf00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:50.083432 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.082931 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff023bfe-ed10-4ea6-bef1-bb220444bf00-kube-api-access-5mfhh" pod "ff023bfe-ed10-4ea6-bef1-bb220444bf00" (UID: "ff023bfe-ed10-4ea6-bef1-bb220444bf00"). InnerVolumeSpecName "kube-api-access-5mfhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:50.085949 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:13:50.085790 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b123ee_6eab_443c_9a71_af860f6f5ace.slice/crio-6fc3f7bc47d626041e7fa4558537ba39c515fefe96cacca3327591ac281ca6e1 WatchSource:0}: Error finding container 6fc3f7bc47d626041e7fa4558537ba39c515fefe96cacca3327591ac281ca6e1: Status 404 returned error can't find the container with id 6fc3f7bc47d626041e7fa4558537ba39c515fefe96cacca3327591ac281ca6e1 Nov 03 21:13:50.098965 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.098932 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dacc6-5d69b9ffff-dwd78" Nov 03 21:13:50.164168 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.164139 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mfhh\" (UniqueName: \"kubernetes.io/projected/ff023bfe-ed10-4ea6-bef1-bb220444bf00-kube-api-access-5mfhh\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:50.164168 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.164165 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff023bfe-ed10-4ea6-bef1-bb220444bf00-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:50.265296 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.265259 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af9e936f-0ebb-449b-b00b-0716f2525039-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af9e936f-0ebb-449b-b00b-0716f2525039-plugins-config-volume\") pod \"af9e936f-0ebb-449b-b00b-0716f2525039\" (UID: \"af9e936f-0ebb-449b-b00b-0716f2525039\") " Nov 03 21:13:50.265489 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.265350 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af9e936f-0ebb-449b-b00b-0716f2525039-kube-api-access-jqtm5\" (UniqueName: \"kubernetes.io/projected/af9e936f-0ebb-449b-b00b-0716f2525039-kube-api-access-jqtm5\") pod \"af9e936f-0ebb-449b-b00b-0716f2525039\" (UID: \"af9e936f-0ebb-449b-b00b-0716f2525039\") " Nov 03 21:13:50.265671 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.265647 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af9e936f-0ebb-449b-b00b-0716f2525039-plugins-config-volume" pod "af9e936f-0ebb-449b-b00b-0716f2525039" (UID: "af9e936f-0ebb-449b-b00b-0716f2525039"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:50.265885 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.265870 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af9e936f-0ebb-449b-b00b-0716f2525039-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:50.267520 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.267499 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9e936f-0ebb-449b-b00b-0716f2525039-kube-api-access-jqtm5" pod "af9e936f-0ebb-449b-b00b-0716f2525039" (UID: "af9e936f-0ebb-449b-b00b-0716f2525039"). InnerVolumeSpecName "kube-api-access-jqtm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:50.367893 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.367809 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqtm5\" (UniqueName: \"kubernetes.io/projected/af9e936f-0ebb-449b-b00b-0716f2525039-kube-api-access-jqtm5\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:50.722265 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.722165 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-6878775966-spjk6" event={"ID":"e0b123ee-6eab-443c-9a71-af860f6f5ace","Type":"ContainerStarted","Data":"39857459fc21a9a05564850762a487267870970f0472af7c476f02c8a2c5405f"} Nov 03 21:13:50.722265 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.722220 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-6878775966-spjk6" event={"ID":"e0b123ee-6eab-443c-9a71-af860f6f5ace","Type":"ContainerStarted","Data":"6fc3f7bc47d626041e7fa4558537ba39c515fefe96cacca3327591ac281ca6e1"} Nov 03 21:13:50.722691 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.722280 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-6878775966-spjk6" Nov 03 21:13:50.723976 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.723946 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-69f498847-h225h" event={"ID":"ce818536-2d2c-4aa1-865d-1921957f995e","Type":"ContainerStarted","Data":"b53b6f461cf984259af3fbae8b9d489038c41a52f930df4d4478582022aa2cfd"} Nov 03 21:13:50.724113 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.724004 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-69f498847-h225h" event={"ID":"ce818536-2d2c-4aa1-865d-1921957f995e","Type":"ContainerStarted","Data":"514f15732b2c5d2c9d4384878849cba3bb2361bedb04607ece36bec7e68ecacc"} Nov 03 21:13:50.724113 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.724029 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-69f498847-h225h" Nov 03 21:13:50.725412 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.725394 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6dacc6-78c86d8fb4-9hmzf" Nov 03 21:13:50.725511 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.725432 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6dacc6-78c86d8fb4-9hmzf" event={"ID":"ff023bfe-ed10-4ea6-bef1-bb220444bf00","Type":"ContainerDied","Data":"b2e0009603a795db533b4024afea4c9e598e9ef08aa58e14bfc29d888f238b59"} Nov 03 21:13:50.725511 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.725481 5615 scope.go:122] "RemoveContainer" containerID="57800c9abf3279ab463a195ade2d8c9a05d256bf77ac58cee3f10ae7d42951f7" Nov 03 21:13:50.726835 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.726814 5615 generic.go:356] "Generic (PLEG): container finished" podID="af9e936f-0ebb-449b-b00b-0716f2525039" containerID="bbe7c825d14d906ac41669677b5b687b3c332e08ff85f0001ed987db8baff451" exitCode=0 Nov 03 21:13:50.726904 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.726844 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dacc6-5d69b9ffff-dwd78" Nov 03 21:13:50.726904 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.726866 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dacc6-5d69b9ffff-dwd78" event={"ID":"af9e936f-0ebb-449b-b00b-0716f2525039","Type":"ContainerDied","Data":"bbe7c825d14d906ac41669677b5b687b3c332e08ff85f0001ed987db8baff451"} Nov 03 21:13:50.726992 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.726901 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dacc6-5d69b9ffff-dwd78" event={"ID":"af9e936f-0ebb-449b-b00b-0716f2525039","Type":"ContainerDied","Data":"d444aab9160bd680fcad52f63c6c48db508be17dabaf97006d346f3fbb39b049"} Nov 03 21:13:50.737055 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.737025 5615 scope.go:122] "RemoveContainer" containerID="bbe7c825d14d906ac41669677b5b687b3c332e08ff85f0001ed987db8baff451" Nov 03 21:13:50.747495 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.747474 5615 scope.go:122] "RemoveContainer" containerID="bbe7c825d14d906ac41669677b5b687b3c332e08ff85f0001ed987db8baff451" Nov 03 21:13:50.747825 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:13:50.747805 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbe7c825d14d906ac41669677b5b687b3c332e08ff85f0001ed987db8baff451\": container with ID starting with bbe7c825d14d906ac41669677b5b687b3c332e08ff85f0001ed987db8baff451 not found: ID does not exist" containerID="bbe7c825d14d906ac41669677b5b687b3c332e08ff85f0001ed987db8baff451" Nov 03 21:13:50.747923 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.747836 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbe7c825d14d906ac41669677b5b687b3c332e08ff85f0001ed987db8baff451"} err="failed to get container status \"bbe7c825d14d906ac41669677b5b687b3c332e08ff85f0001ed987db8baff451\": rpc error: code = NotFound desc = could not find container \"bbe7c825d14d906ac41669677b5b687b3c332e08ff85f0001ed987db8baff451\": container with ID starting with bbe7c825d14d906ac41669677b5b687b3c332e08ff85f0001ed987db8baff451 not found: ID does not exist" Nov 03 21:13:50.843170 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.843119 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-6878775966-spjk6" podStartSLOduration=2.843103007 podStartE2EDuration="2.843103007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:13:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:13:50.841566642 +0000 UTC m=+1525.309345369" watchObservedRunningTime="2026-11-03 21:13:50.843103007 +0000 UTC m=+1525.310881732" Nov 03 21:13:50.949095 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.949048 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6dacc6-5d69b9ffff-dwd78"] Nov 03 21:13:50.952330 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:50.952294 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6dacc6-5d69b9ffff-dwd78"] Nov 03 21:13:51.069027 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:51.068979 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6dacc6-78c86d8fb4-9hmzf"] Nov 03 21:13:51.080146 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:51.080069 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6dacc6-78c86d8fb4-9hmzf"] Nov 03 21:13:51.179904 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:51.179851 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-69f498847-h225h" podStartSLOduration=3.179837326 podStartE2EDuration="3.179837326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:13:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:13:51.178714152 +0000 UTC m=+1525.646492887" watchObservedRunningTime="2026-11-03 21:13:51.179837326 +0000 UTC m=+1525.647616049" Nov 03 21:13:51.653357 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:51.653318 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af9e936f-0ebb-449b-b00b-0716f2525039" path="/var/lib/kubelet/pods/af9e936f-0ebb-449b-b00b-0716f2525039/volumes" Nov 03 21:13:51.653861 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:51.653846 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff023bfe-ed10-4ea6-bef1-bb220444bf00" path="/var/lib/kubelet/pods/ff023bfe-ed10-4ea6-bef1-bb220444bf00/volumes" Nov 03 21:13:52.105823 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:52.105787 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6ad223-577d847bf6-krjwf"] Nov 03 21:13:52.215992 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:52.215959 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6ad223-56556d55ff-qnczx"] Nov 03 21:13:52.606097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:52.604915 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-22600c-68c974ccf4-kp5mr" Nov 03 21:13:57.106625 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.106575 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-6ad223-577d847bf6-krjwf" podUID="589203b0-97f0-4e72-a1e3-091039e5d0ce" containerName="payload-processing" containerID="cri-o://5056c22ccdf4de15a414dae54d6fe19a533e00027ec07cae5d30f3d03301fb60" gracePeriod=25 Nov 03 21:13:57.216509 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.216460 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ad223-56556d55ff-qnczx" podUID="a07203b7-b82b-4329-9c71-f50ee6e77374" containerName="payload-pre-processing" containerID="cri-o://7b4bcf59e7dccbd98e3641faf0b4ea2369387e829deb058e2f1827dda3314778" gracePeriod=25 Nov 03 21:13:57.408208 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.407788 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6ad223-577d847bf6-krjwf" Nov 03 21:13:57.501674 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.501651 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ad223-56556d55ff-qnczx" Nov 03 21:13:57.521269 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.521210 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/589203b0-97f0-4e72-a1e3-091039e5d0ce-kube-api-access-dd52s\" (UniqueName: \"kubernetes.io/projected/589203b0-97f0-4e72-a1e3-091039e5d0ce-kube-api-access-dd52s\") pod \"589203b0-97f0-4e72-a1e3-091039e5d0ce\" (UID: \"589203b0-97f0-4e72-a1e3-091039e5d0ce\") " Nov 03 21:13:57.521449 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.521380 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/589203b0-97f0-4e72-a1e3-091039e5d0ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589203b0-97f0-4e72-a1e3-091039e5d0ce-plugins-config-volume\") pod \"589203b0-97f0-4e72-a1e3-091039e5d0ce\" (UID: \"589203b0-97f0-4e72-a1e3-091039e5d0ce\") " Nov 03 21:13:57.522002 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.521938 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/589203b0-97f0-4e72-a1e3-091039e5d0ce-plugins-config-volume" pod "589203b0-97f0-4e72-a1e3-091039e5d0ce" (UID: "589203b0-97f0-4e72-a1e3-091039e5d0ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:57.523981 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.523949 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/589203b0-97f0-4e72-a1e3-091039e5d0ce-kube-api-access-dd52s" pod "589203b0-97f0-4e72-a1e3-091039e5d0ce" (UID: "589203b0-97f0-4e72-a1e3-091039e5d0ce"). InnerVolumeSpecName "kube-api-access-dd52s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:57.623310 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.623272 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a07203b7-b82b-4329-9c71-f50ee6e77374-kube-api-access-zz4kw\" (UniqueName: \"kubernetes.io/projected/a07203b7-b82b-4329-9c71-f50ee6e77374-kube-api-access-zz4kw\") pod \"a07203b7-b82b-4329-9c71-f50ee6e77374\" (UID: \"a07203b7-b82b-4329-9c71-f50ee6e77374\") " Nov 03 21:13:57.623574 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.623555 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a07203b7-b82b-4329-9c71-f50ee6e77374-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a07203b7-b82b-4329-9c71-f50ee6e77374-plugins-config-volume\") pod \"a07203b7-b82b-4329-9c71-f50ee6e77374\" (UID: \"a07203b7-b82b-4329-9c71-f50ee6e77374\") " Nov 03 21:13:57.623933 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.623907 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a07203b7-b82b-4329-9c71-f50ee6e77374-plugins-config-volume" pod "a07203b7-b82b-4329-9c71-f50ee6e77374" (UID: "a07203b7-b82b-4329-9c71-f50ee6e77374"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:57.624234 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.624207 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589203b0-97f0-4e72-a1e3-091039e5d0ce-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:57.624234 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.624234 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a07203b7-b82b-4329-9c71-f50ee6e77374-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:57.624370 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.624247 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd52s\" (UniqueName: \"kubernetes.io/projected/589203b0-97f0-4e72-a1e3-091039e5d0ce-kube-api-access-dd52s\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:57.625608 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.625582 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07203b7-b82b-4329-9c71-f50ee6e77374-kube-api-access-zz4kw" pod "a07203b7-b82b-4329-9c71-f50ee6e77374" (UID: "a07203b7-b82b-4329-9c71-f50ee6e77374"). InnerVolumeSpecName "kube-api-access-zz4kw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:57.725716 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.725676 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz4kw\" (UniqueName: \"kubernetes.io/projected/a07203b7-b82b-4329-9c71-f50ee6e77374-kube-api-access-zz4kw\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:57.833681 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.833641 5615 generic.go:356] "Generic (PLEG): container finished" podID="589203b0-97f0-4e72-a1e3-091039e5d0ce" containerID="5056c22ccdf4de15a414dae54d6fe19a533e00027ec07cae5d30f3d03301fb60" exitCode=0 Nov 03 21:13:57.833879 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.833688 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6ad223-577d847bf6-krjwf" Nov 03 21:13:57.833879 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.833728 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6ad223-577d847bf6-krjwf" event={"ID":"589203b0-97f0-4e72-a1e3-091039e5d0ce","Type":"ContainerDied","Data":"5056c22ccdf4de15a414dae54d6fe19a533e00027ec07cae5d30f3d03301fb60"} Nov 03 21:13:57.833879 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.833783 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6ad223-577d847bf6-krjwf" event={"ID":"589203b0-97f0-4e72-a1e3-091039e5d0ce","Type":"ContainerDied","Data":"f02e8875ee61608748808c205bc4efb7ae8822c8239b389365b8ea987a0e6904"} Nov 03 21:13:57.833879 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.833811 5615 scope.go:122] "RemoveContainer" containerID="5056c22ccdf4de15a414dae54d6fe19a533e00027ec07cae5d30f3d03301fb60" Nov 03 21:13:57.835396 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.835366 5615 generic.go:356] "Generic (PLEG): container finished" podID="a07203b7-b82b-4329-9c71-f50ee6e77374" containerID="7b4bcf59e7dccbd98e3641faf0b4ea2369387e829deb058e2f1827dda3314778" exitCode=0 Nov 03 21:13:57.835506 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.835418 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ad223-56556d55ff-qnczx" Nov 03 21:13:57.835506 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.835455 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ad223-56556d55ff-qnczx" event={"ID":"a07203b7-b82b-4329-9c71-f50ee6e77374","Type":"ContainerDied","Data":"7b4bcf59e7dccbd98e3641faf0b4ea2369387e829deb058e2f1827dda3314778"} Nov 03 21:13:57.835582 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.835512 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ad223-56556d55ff-qnczx" event={"ID":"a07203b7-b82b-4329-9c71-f50ee6e77374","Type":"ContainerDied","Data":"06fa30c11f583132f7a3dce65f02805dc3532648cc7aa942b902ca47134b7eb2"} Nov 03 21:13:57.846549 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.846521 5615 scope.go:122] "RemoveContainer" containerID="5056c22ccdf4de15a414dae54d6fe19a533e00027ec07cae5d30f3d03301fb60" Nov 03 21:13:57.846882 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:13:57.846850 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5056c22ccdf4de15a414dae54d6fe19a533e00027ec07cae5d30f3d03301fb60\": container with ID starting with 5056c22ccdf4de15a414dae54d6fe19a533e00027ec07cae5d30f3d03301fb60 not found: ID does not exist" containerID="5056c22ccdf4de15a414dae54d6fe19a533e00027ec07cae5d30f3d03301fb60" Nov 03 21:13:57.846980 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.846893 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5056c22ccdf4de15a414dae54d6fe19a533e00027ec07cae5d30f3d03301fb60"} err="failed to get container status \"5056c22ccdf4de15a414dae54d6fe19a533e00027ec07cae5d30f3d03301fb60\": rpc error: code = NotFound desc = could not find container \"5056c22ccdf4de15a414dae54d6fe19a533e00027ec07cae5d30f3d03301fb60\": container with ID starting with 5056c22ccdf4de15a414dae54d6fe19a533e00027ec07cae5d30f3d03301fb60 not found: ID does not exist" Nov 03 21:13:57.846980 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.846918 5615 scope.go:122] "RemoveContainer" containerID="7b4bcf59e7dccbd98e3641faf0b4ea2369387e829deb058e2f1827dda3314778" Nov 03 21:13:57.856816 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.856792 5615 scope.go:122] "RemoveContainer" containerID="7b4bcf59e7dccbd98e3641faf0b4ea2369387e829deb058e2f1827dda3314778" Nov 03 21:13:57.857120 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:13:57.857095 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b4bcf59e7dccbd98e3641faf0b4ea2369387e829deb058e2f1827dda3314778\": container with ID starting with 7b4bcf59e7dccbd98e3641faf0b4ea2369387e829deb058e2f1827dda3314778 not found: ID does not exist" containerID="7b4bcf59e7dccbd98e3641faf0b4ea2369387e829deb058e2f1827dda3314778" Nov 03 21:13:57.857190 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.857129 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b4bcf59e7dccbd98e3641faf0b4ea2369387e829deb058e2f1827dda3314778"} err="failed to get container status \"7b4bcf59e7dccbd98e3641faf0b4ea2369387e829deb058e2f1827dda3314778\": rpc error: code = NotFound desc = could not find container \"7b4bcf59e7dccbd98e3641faf0b4ea2369387e829deb058e2f1827dda3314778\": container with ID starting with 7b4bcf59e7dccbd98e3641faf0b4ea2369387e829deb058e2f1827dda3314778 not found: ID does not exist" Nov 03 21:13:57.991611 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.991578 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6ad223-577d847bf6-krjwf"] Nov 03 21:13:57.996279 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:57.996253 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6ad223-577d847bf6-krjwf"] Nov 03 21:13:58.099917 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:58.099883 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6ad223-56556d55ff-qnczx"] Nov 03 21:13:58.103779 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:58.103745 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6ad223-56556d55ff-qnczx"] Nov 03 21:13:58.563796 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:58.563759 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw"] Nov 03 21:13:58.564214 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:58.563961 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw" podUID="5baca77b-0b26-4236-ad3b-c0102e26b1eb" containerName="maas-api" containerID="cri-o://14de179b55d9ad9462d48d80e56a01f55cae8597295ab6d451fc4c5116c5f3ee" gracePeriod=30 Nov 03 21:13:58.836638 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:58.836610 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw" Nov 03 21:13:58.843757 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:58.843722 5615 generic.go:356] "Generic (PLEG): container finished" podID="5baca77b-0b26-4236-ad3b-c0102e26b1eb" containerID="14de179b55d9ad9462d48d80e56a01f55cae8597295ab6d451fc4c5116c5f3ee" exitCode=0 Nov 03 21:13:58.843938 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:58.843771 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw" Nov 03 21:13:58.843938 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:58.843801 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw" event={"ID":"5baca77b-0b26-4236-ad3b-c0102e26b1eb","Type":"ContainerDied","Data":"14de179b55d9ad9462d48d80e56a01f55cae8597295ab6d451fc4c5116c5f3ee"} Nov 03 21:13:58.843938 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:58.843866 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw" event={"ID":"5baca77b-0b26-4236-ad3b-c0102e26b1eb","Type":"ContainerDied","Data":"1ce78b99b9f27906cf96a55f27b1fbd61d6004ee6d570a87fabd068963de1734"} Nov 03 21:13:58.843938 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:58.843893 5615 scope.go:122] "RemoveContainer" containerID="14de179b55d9ad9462d48d80e56a01f55cae8597295ab6d451fc4c5116c5f3ee" Nov 03 21:13:58.905019 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:58.904912 5615 scope.go:122] "RemoveContainer" containerID="14de179b55d9ad9462d48d80e56a01f55cae8597295ab6d451fc4c5116c5f3ee" Nov 03 21:13:58.912140 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:13:58.905847 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14de179b55d9ad9462d48d80e56a01f55cae8597295ab6d451fc4c5116c5f3ee\": container with ID starting with 14de179b55d9ad9462d48d80e56a01f55cae8597295ab6d451fc4c5116c5f3ee not found: ID does not exist" containerID="14de179b55d9ad9462d48d80e56a01f55cae8597295ab6d451fc4c5116c5f3ee" Nov 03 21:13:58.912140 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:58.905902 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14de179b55d9ad9462d48d80e56a01f55cae8597295ab6d451fc4c5116c5f3ee"} err="failed to get container status \"14de179b55d9ad9462d48d80e56a01f55cae8597295ab6d451fc4c5116c5f3ee\": rpc error: code = NotFound desc = could not find container \"14de179b55d9ad9462d48d80e56a01f55cae8597295ab6d451fc4c5116c5f3ee\": container with ID starting with 14de179b55d9ad9462d48d80e56a01f55cae8597295ab6d451fc4c5116c5f3ee not found: ID does not exist" Nov 03 21:13:58.959101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:58.946656 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5baca77b-0b26-4236-ad3b-c0102e26b1eb-kube-api-access-kvvwt\" (UniqueName: \"kubernetes.io/projected/5baca77b-0b26-4236-ad3b-c0102e26b1eb-kube-api-access-kvvwt\") pod \"5baca77b-0b26-4236-ad3b-c0102e26b1eb\" (UID: \"5baca77b-0b26-4236-ad3b-c0102e26b1eb\") " Nov 03 21:13:58.959101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:58.946811 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5baca77b-0b26-4236-ad3b-c0102e26b1eb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5baca77b-0b26-4236-ad3b-c0102e26b1eb-metrics-certs\") pod \"5baca77b-0b26-4236-ad3b-c0102e26b1eb\" (UID: \"5baca77b-0b26-4236-ad3b-c0102e26b1eb\") " Nov 03 21:13:58.959101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:58.946845 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5baca77b-0b26-4236-ad3b-c0102e26b1eb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5baca77b-0b26-4236-ad3b-c0102e26b1eb-openshift-service-ca\") pod \"5baca77b-0b26-4236-ad3b-c0102e26b1eb\" (UID: \"5baca77b-0b26-4236-ad3b-c0102e26b1eb\") " Nov 03 21:13:58.959101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:58.947030 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5baca77b-0b26-4236-ad3b-c0102e26b1eb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5baca77b-0b26-4236-ad3b-c0102e26b1eb-maas-api-tls\") pod \"5baca77b-0b26-4236-ad3b-c0102e26b1eb\" (UID: \"5baca77b-0b26-4236-ad3b-c0102e26b1eb\") " Nov 03 21:13:58.979098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:58.962593 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5baca77b-0b26-4236-ad3b-c0102e26b1eb-openshift-service-ca" pod "5baca77b-0b26-4236-ad3b-c0102e26b1eb" (UID: "5baca77b-0b26-4236-ad3b-c0102e26b1eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:58.989284 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:58.989222 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5baca77b-0b26-4236-ad3b-c0102e26b1eb-maas-api-tls" pod "5baca77b-0b26-4236-ad3b-c0102e26b1eb" (UID: "5baca77b-0b26-4236-ad3b-c0102e26b1eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:13:58.991129 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:58.991020 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5baca77b-0b26-4236-ad3b-c0102e26b1eb-metrics-certs" pod "5baca77b-0b26-4236-ad3b-c0102e26b1eb" (UID: "5baca77b-0b26-4236-ad3b-c0102e26b1eb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:13:58.991279 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:58.991194 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5baca77b-0b26-4236-ad3b-c0102e26b1eb-kube-api-access-kvvwt" pod "5baca77b-0b26-4236-ad3b-c0102e26b1eb" (UID: "5baca77b-0b26-4236-ad3b-c0102e26b1eb"). InnerVolumeSpecName "kube-api-access-kvvwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:59.048952 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:59.048914 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvvwt\" (UniqueName: \"kubernetes.io/projected/5baca77b-0b26-4236-ad3b-c0102e26b1eb-kube-api-access-kvvwt\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:59.048952 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:59.048947 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5baca77b-0b26-4236-ad3b-c0102e26b1eb-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:59.048952 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:59.048958 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5baca77b-0b26-4236-ad3b-c0102e26b1eb-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:59.049204 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:59.048967 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5baca77b-0b26-4236-ad3b-c0102e26b1eb-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:59.295790 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:59.295744 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw"] Nov 03 21:13:59.303925 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:59.303883 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6dacc6-897d59d4d-2qqjw"] Nov 03 21:13:59.677101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:59.670874 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="589203b0-97f0-4e72-a1e3-091039e5d0ce" path="/var/lib/kubelet/pods/589203b0-97f0-4e72-a1e3-091039e5d0ce/volumes" Nov 03 21:13:59.677101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:59.671585 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5baca77b-0b26-4236-ad3b-c0102e26b1eb" path="/var/lib/kubelet/pods/5baca77b-0b26-4236-ad3b-c0102e26b1eb/volumes" Nov 03 21:13:59.677101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:13:59.672186 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a07203b7-b82b-4329-9c71-f50ee6e77374" path="/var/lib/kubelet/pods/a07203b7-b82b-4329-9c71-f50ee6e77374/volumes" Nov 03 21:14:01.739163 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:01.739129 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-69f498847-h225h" Nov 03 21:14:01.739619 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:01.739194 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-6878775966-spjk6" Nov 03 21:14:10.620456 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:14:10.620371 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:14:16.047456 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:16.047419 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3337df-6897659f76-nt5hq"] Nov 03 21:14:19.165342 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.165305 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz"] Nov 03 21:14:19.165979 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.165963 5615 cpu_manager.go:414] "containerMap: removing container" podUID="a07203b7-b82b-4329-9c71-f50ee6e77374" containerName="payload-pre-processing" Nov 03 21:14:19.166024 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.165980 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a07203b7-b82b-4329-9c71-f50ee6e77374" containerName="payload-pre-processing" Nov 03 21:14:19.166024 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.166003 5615 cpu_manager.go:414] "containerMap: removing container" podUID="af9e936f-0ebb-449b-b00b-0716f2525039" containerName="payload-pre-processing" Nov 03 21:14:19.166024 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.166008 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af9e936f-0ebb-449b-b00b-0716f2525039" containerName="payload-pre-processing" Nov 03 21:14:19.166024 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.166023 5615 cpu_manager.go:414] "containerMap: removing container" podUID="5baca77b-0b26-4236-ad3b-c0102e26b1eb" containerName="maas-api" Nov 03 21:14:19.166163 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.166028 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5baca77b-0b26-4236-ad3b-c0102e26b1eb" containerName="maas-api" Nov 03 21:14:19.166163 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.166047 5615 cpu_manager.go:414] "containerMap: removing container" podUID="ff023bfe-ed10-4ea6-bef1-bb220444bf00" containerName="payload-processing" Nov 03 21:14:19.166163 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.166052 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff023bfe-ed10-4ea6-bef1-bb220444bf00" containerName="payload-processing" Nov 03 21:14:19.166163 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.166068 5615 cpu_manager.go:414] "containerMap: removing container" podUID="589203b0-97f0-4e72-a1e3-091039e5d0ce" containerName="payload-processing" Nov 03 21:14:19.166163 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.166087 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="589203b0-97f0-4e72-a1e3-091039e5d0ce" containerName="payload-processing" Nov 03 21:14:19.166311 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.166180 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="589203b0-97f0-4e72-a1e3-091039e5d0ce" containerName="payload-processing" Nov 03 21:14:19.166311 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.166191 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="a07203b7-b82b-4329-9c71-f50ee6e77374" containerName="payload-pre-processing" Nov 03 21:14:19.166311 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.166200 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="af9e936f-0ebb-449b-b00b-0716f2525039" containerName="payload-pre-processing" Nov 03 21:14:19.166311 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.166207 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="5baca77b-0b26-4236-ad3b-c0102e26b1eb" containerName="maas-api" Nov 03 21:14:19.166311 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.166216 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff023bfe-ed10-4ea6-bef1-bb220444bf00" containerName="payload-processing" Nov 03 21:14:19.169831 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.169811 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz" Nov 03 21:14:19.268697 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.268655 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-22600c-serving-cert\"" Nov 03 21:14:19.271389 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.271353 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-openshift-service-ca\") pod \"maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz\" (UID: \"86bccd8d-d5c7-4e37-8a69-334f7b1bdd48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz" Nov 03 21:14:19.271535 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.271419 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-metrics-certs\") pod \"maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz\" (UID: \"86bccd8d-d5c7-4e37-8a69-334f7b1bdd48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz" Nov 03 21:14:19.271535 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.271517 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-maas-api-tls\") pod \"maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz\" (UID: \"86bccd8d-d5c7-4e37-8a69-334f7b1bdd48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz" Nov 03 21:14:19.271647 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.271630 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zqgw\" (UniqueName: \"kubernetes.io/projected/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-kube-api-access-7zqgw\") pod \"maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz\" (UID: \"86bccd8d-d5c7-4e37-8a69-334f7b1bdd48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz" Nov 03 21:14:19.273427 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.273402 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz"] Nov 03 21:14:19.373161 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.373120 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-maas-api-tls\") pod \"maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz\" (UID: \"86bccd8d-d5c7-4e37-8a69-334f7b1bdd48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz" Nov 03 21:14:19.373349 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.373229 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zqgw\" (UniqueName: \"kubernetes.io/projected/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-kube-api-access-7zqgw\") pod \"maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz\" (UID: \"86bccd8d-d5c7-4e37-8a69-334f7b1bdd48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz" Nov 03 21:14:19.373349 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:14:19.373273 5615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-serving-cert: secret "maas-api-e2e-api-b-22600c-serving-cert" not found Nov 03 21:14:19.373349 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:14:19.373342 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-maas-api-tls podName:86bccd8d-d5c7-4e37-8a69-334f7b1bdd48 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:19.873324027 +0000 UTC m=+1554.341102729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-maas-api-tls") pod "maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz" (UID: "86bccd8d-d5c7-4e37-8a69-334f7b1bdd48") : secret "maas-api-e2e-api-b-22600c-serving-cert" not found Nov 03 21:14:19.373484 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.373454 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-openshift-service-ca\") pod \"maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz\" (UID: \"86bccd8d-d5c7-4e37-8a69-334f7b1bdd48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz" Nov 03 21:14:19.373529 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.373511 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-metrics-certs\") pod \"maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz\" (UID: \"86bccd8d-d5c7-4e37-8a69-334f7b1bdd48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz" Nov 03 21:14:19.374149 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.374125 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-openshift-service-ca\") pod \"maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz\" (UID: \"86bccd8d-d5c7-4e37-8a69-334f7b1bdd48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz" Nov 03 21:14:19.376259 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.376232 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-metrics-certs\") pod \"maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz\" (UID: \"86bccd8d-d5c7-4e37-8a69-334f7b1bdd48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz" Nov 03 21:14:19.468905 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.468827 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zqgw\" (UniqueName: \"kubernetes.io/projected/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-kube-api-access-7zqgw\") pod \"maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz\" (UID: \"86bccd8d-d5c7-4e37-8a69-334f7b1bdd48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz" Nov 03 21:14:19.893957 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.893911 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-maas-api-tls\") pod \"maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz\" (UID: \"86bccd8d-d5c7-4e37-8a69-334f7b1bdd48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz" Nov 03 21:14:19.896445 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:19.896418 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-maas-api-tls\") pod \"maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz\" (UID: \"86bccd8d-d5c7-4e37-8a69-334f7b1bdd48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz" Nov 03 21:14:20.084304 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:20.084272 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz" Nov 03 21:14:20.460829 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:20.460800 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz"] Nov 03 21:14:20.462538 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:14:20.462499 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86bccd8d_d5c7_4e37_8a69_334f7b1bdd48.slice/crio-0e1a798475a745d40c0071f8b04d9da4c41a54f40cc2cb7422d375117230acc2 WatchSource:0}: Error finding container 0e1a798475a745d40c0071f8b04d9da4c41a54f40cc2cb7422d375117230acc2: Status 404 returned error can't find the container with id 0e1a798475a745d40c0071f8b04d9da4c41a54f40cc2cb7422d375117230acc2 Nov 03 21:14:21.019338 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:21.019298 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz" event={"ID":"86bccd8d-d5c7-4e37-8a69-334f7b1bdd48","Type":"ContainerStarted","Data":"0e1a798475a745d40c0071f8b04d9da4c41a54f40cc2cb7422d375117230acc2"} Nov 03 21:14:21.048766 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:21.048719 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-3337df-6897659f76-nt5hq" podUID="2e6932b4-504a-4fcf-8d86-0fda75a23e6b" containerName="payload-processing" containerID="cri-o://95e04ad8f281e383a49d86ee8377e198e5e98e2c6707218547fe3f19da109e65" gracePeriod=25 Nov 03 21:14:21.397122 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:21.397096 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3337df-6897659f76-nt5hq" Nov 03 21:14:21.552479 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:21.552442 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e6932b4-504a-4fcf-8d86-0fda75a23e6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e6932b4-504a-4fcf-8d86-0fda75a23e6b-plugins-config-volume\") pod \"2e6932b4-504a-4fcf-8d86-0fda75a23e6b\" (UID: \"2e6932b4-504a-4fcf-8d86-0fda75a23e6b\") " Nov 03 21:14:21.552893 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:21.552488 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e6932b4-504a-4fcf-8d86-0fda75a23e6b-kube-api-access-s7hgf\" (UniqueName: \"kubernetes.io/projected/2e6932b4-504a-4fcf-8d86-0fda75a23e6b-kube-api-access-s7hgf\") pod \"2e6932b4-504a-4fcf-8d86-0fda75a23e6b\" (UID: \"2e6932b4-504a-4fcf-8d86-0fda75a23e6b\") " Nov 03 21:14:21.552966 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:21.552897 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e6932b4-504a-4fcf-8d86-0fda75a23e6b-plugins-config-volume" pod "2e6932b4-504a-4fcf-8d86-0fda75a23e6b" (UID: "2e6932b4-504a-4fcf-8d86-0fda75a23e6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:21.553349 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:21.553310 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e6932b4-504a-4fcf-8d86-0fda75a23e6b-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:21.555190 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:21.555156 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e6932b4-504a-4fcf-8d86-0fda75a23e6b-kube-api-access-s7hgf" pod "2e6932b4-504a-4fcf-8d86-0fda75a23e6b" (UID: "2e6932b4-504a-4fcf-8d86-0fda75a23e6b"). InnerVolumeSpecName "kube-api-access-s7hgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:21.654323 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:21.654235 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7hgf\" (UniqueName: \"kubernetes.io/projected/2e6932b4-504a-4fcf-8d86-0fda75a23e6b-kube-api-access-s7hgf\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:22.027990 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:22.027957 5615 generic.go:356] "Generic (PLEG): container finished" podID="2e6932b4-504a-4fcf-8d86-0fda75a23e6b" containerID="95e04ad8f281e383a49d86ee8377e198e5e98e2c6707218547fe3f19da109e65" exitCode=0 Nov 03 21:14:22.028181 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:22.028004 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3337df-6897659f76-nt5hq" Nov 03 21:14:22.028181 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:22.028033 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3337df-6897659f76-nt5hq" event={"ID":"2e6932b4-504a-4fcf-8d86-0fda75a23e6b","Type":"ContainerDied","Data":"95e04ad8f281e383a49d86ee8377e198e5e98e2c6707218547fe3f19da109e65"} Nov 03 21:14:22.028181 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:22.028120 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3337df-6897659f76-nt5hq" event={"ID":"2e6932b4-504a-4fcf-8d86-0fda75a23e6b","Type":"ContainerDied","Data":"5223e6894306bc59ce5665ed91164597be1a0f79e36d777f7391b3ccdf6ddc2c"} Nov 03 21:14:22.028181 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:22.028149 5615 scope.go:122] "RemoveContainer" containerID="95e04ad8f281e383a49d86ee8377e198e5e98e2c6707218547fe3f19da109e65" Nov 03 21:14:22.030316 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:22.030287 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz" event={"ID":"86bccd8d-d5c7-4e37-8a69-334f7b1bdd48","Type":"ContainerStarted","Data":"df9b88913f3d02fdfa61bd7c292e2e8441f0747c4ab0e0c528e763f8886c608f"} Nov 03 21:14:22.030569 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:22.030553 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz" Nov 03 21:14:22.039483 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:22.039463 5615 scope.go:122] "RemoveContainer" containerID="95e04ad8f281e383a49d86ee8377e198e5e98e2c6707218547fe3f19da109e65" Nov 03 21:14:22.039804 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:14:22.039752 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95e04ad8f281e383a49d86ee8377e198e5e98e2c6707218547fe3f19da109e65\": container with ID starting with 95e04ad8f281e383a49d86ee8377e198e5e98e2c6707218547fe3f19da109e65 not found: ID does not exist" containerID="95e04ad8f281e383a49d86ee8377e198e5e98e2c6707218547fe3f19da109e65" Nov 03 21:14:22.039804 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:22.039791 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e04ad8f281e383a49d86ee8377e198e5e98e2c6707218547fe3f19da109e65"} err="failed to get container status \"95e04ad8f281e383a49d86ee8377e198e5e98e2c6707218547fe3f19da109e65\": rpc error: code = NotFound desc = could not find container \"95e04ad8f281e383a49d86ee8377e198e5e98e2c6707218547fe3f19da109e65\": container with ID starting with 95e04ad8f281e383a49d86ee8377e198e5e98e2c6707218547fe3f19da109e65 not found: ID does not exist" Nov 03 21:14:22.215248 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:22.215108 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz" podStartSLOduration=2.245784481 podStartE2EDuration="3.215086875s" totalImagesPullingTime="969.302394ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:14:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:22.212358887 +0000 UTC m=+1556.680137644" watchObservedRunningTime="2026-11-03 21:14:22.215086875 +0000 UTC m=+1556.682865590" Nov 03 21:14:22.318853 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:22.318768 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3337df-6897659f76-nt5hq"] Nov 03 21:14:22.326726 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:22.326691 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3337df-6897659f76-nt5hq"] Nov 03 21:14:23.652749 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:23.652711 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e6932b4-504a-4fcf-8d86-0fda75a23e6b" path="/var/lib/kubelet/pods/2e6932b4-504a-4fcf-8d86-0fda75a23e6b/volumes" Nov 03 21:14:27.928961 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:27.928926 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4"] Nov 03 21:14:27.929452 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:27.929150 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4" podUID="04594780-d2d0-4b95-85bf-db2578d8876f" containerName="maas-api" containerID="cri-o://d5b54611fa75ce70c6fa1b2991cd76926cf22e7bb2b3122e49dc2cd6d8de9522" gracePeriod=30 Nov 03 21:14:28.045498 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:28.045466 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz" Nov 03 21:14:28.074629 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:28.074596 5615 generic.go:356] "Generic (PLEG): container finished" podID="04594780-d2d0-4b95-85bf-db2578d8876f" containerID="d5b54611fa75ce70c6fa1b2991cd76926cf22e7bb2b3122e49dc2cd6d8de9522" exitCode=0 Nov 03 21:14:28.074802 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:28.074665 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4" event={"ID":"04594780-d2d0-4b95-85bf-db2578d8876f","Type":"ContainerDied","Data":"d5b54611fa75ce70c6fa1b2991cd76926cf22e7bb2b3122e49dc2cd6d8de9522"} Nov 03 21:14:28.319273 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:28.319242 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4" Nov 03 21:14:28.410103 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:28.406605 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04594780-d2d0-4b95-85bf-db2578d8876f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04594780-d2d0-4b95-85bf-db2578d8876f-maas-api-tls\") pod \"04594780-d2d0-4b95-85bf-db2578d8876f\" (UID: \"04594780-d2d0-4b95-85bf-db2578d8876f\") " Nov 03 21:14:28.410103 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:28.406693 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04594780-d2d0-4b95-85bf-db2578d8876f-kube-api-access-z56n4\" (UniqueName: \"kubernetes.io/projected/04594780-d2d0-4b95-85bf-db2578d8876f-kube-api-access-z56n4\") pod \"04594780-d2d0-4b95-85bf-db2578d8876f\" (UID: \"04594780-d2d0-4b95-85bf-db2578d8876f\") " Nov 03 21:14:28.410103 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:28.406967 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04594780-d2d0-4b95-85bf-db2578d8876f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/04594780-d2d0-4b95-85bf-db2578d8876f-metrics-certs\") pod \"04594780-d2d0-4b95-85bf-db2578d8876f\" (UID: \"04594780-d2d0-4b95-85bf-db2578d8876f\") " Nov 03 21:14:28.410103 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:28.407027 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04594780-d2d0-4b95-85bf-db2578d8876f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04594780-d2d0-4b95-85bf-db2578d8876f-openshift-service-ca\") pod \"04594780-d2d0-4b95-85bf-db2578d8876f\" (UID: \"04594780-d2d0-4b95-85bf-db2578d8876f\") " Nov 03 21:14:28.426100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:28.419462 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04594780-d2d0-4b95-85bf-db2578d8876f-openshift-service-ca" pod "04594780-d2d0-4b95-85bf-db2578d8876f" (UID: "04594780-d2d0-4b95-85bf-db2578d8876f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:28.486108 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:28.483313 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04594780-d2d0-4b95-85bf-db2578d8876f-metrics-certs" pod "04594780-d2d0-4b95-85bf-db2578d8876f" (UID: "04594780-d2d0-4b95-85bf-db2578d8876f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:14:28.494106 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:28.490212 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04594780-d2d0-4b95-85bf-db2578d8876f-maas-api-tls" pod "04594780-d2d0-4b95-85bf-db2578d8876f" (UID: "04594780-d2d0-4b95-85bf-db2578d8876f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:14:28.512645 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:28.510469 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04594780-d2d0-4b95-85bf-db2578d8876f-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:28.512645 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:28.510511 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04594780-d2d0-4b95-85bf-db2578d8876f-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:28.512645 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:28.510527 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04594780-d2d0-4b95-85bf-db2578d8876f-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:28.512645 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:28.511248 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04594780-d2d0-4b95-85bf-db2578d8876f-kube-api-access-z56n4" pod "04594780-d2d0-4b95-85bf-db2578d8876f" (UID: "04594780-d2d0-4b95-85bf-db2578d8876f"). InnerVolumeSpecName "kube-api-access-z56n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:28.612190 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:28.612085 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z56n4\" (UniqueName: \"kubernetes.io/projected/04594780-d2d0-4b95-85bf-db2578d8876f-kube-api-access-z56n4\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:29.082492 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:29.082452 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4" Nov 03 21:14:29.082982 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:29.082484 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4" event={"ID":"04594780-d2d0-4b95-85bf-db2578d8876f","Type":"ContainerDied","Data":"5167c5edf7917cc7f4383cfe07b15df832ea2d3285723695f9171f402fba5260"} Nov 03 21:14:29.082982 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:29.082554 5615 scope.go:122] "RemoveContainer" containerID="d5b54611fa75ce70c6fa1b2991cd76926cf22e7bb2b3122e49dc2cd6d8de9522" Nov 03 21:14:29.240264 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:29.240232 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4"] Nov 03 21:14:29.287357 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:29.287326 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-4l5s4"] Nov 03 21:14:29.653663 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:29.653633 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04594780-d2d0-4b95-85bf-db2578d8876f" path="/var/lib/kubelet/pods/04594780-d2d0-4b95-85bf-db2578d8876f/volumes" Nov 03 21:14:34.272686 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:34.272650 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm"] Nov 03 21:14:34.273378 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:34.273363 5615 cpu_manager.go:414] "containerMap: removing container" podUID="2e6932b4-504a-4fcf-8d86-0fda75a23e6b" containerName="payload-processing" Nov 03 21:14:34.273430 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:34.273380 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e6932b4-504a-4fcf-8d86-0fda75a23e6b" containerName="payload-processing" Nov 03 21:14:34.273430 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:34.273393 5615 cpu_manager.go:414] "containerMap: removing container" podUID="04594780-d2d0-4b95-85bf-db2578d8876f" containerName="maas-api" Nov 03 21:14:34.273430 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:34.273399 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04594780-d2d0-4b95-85bf-db2578d8876f" containerName="maas-api" Nov 03 21:14:34.273518 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:34.273494 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e6932b4-504a-4fcf-8d86-0fda75a23e6b" containerName="payload-processing" Nov 03 21:14:34.273518 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:34.273503 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="04594780-d2d0-4b95-85bf-db2578d8876f" containerName="maas-api" Nov 03 21:14:34.278576 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:34.278552 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm" Nov 03 21:14:34.309362 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:34.309326 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw9qh\" (UniqueName: \"kubernetes.io/projected/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-kube-api-access-xw9qh\") pod \"maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm\" (UID: \"c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm" Nov 03 21:14:34.309542 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:34.309477 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm\" (UID: \"c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm" Nov 03 21:14:34.309542 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:34.309534 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm\" (UID: \"c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm" Nov 03 21:14:34.309655 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:34.309563 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-metrics-certs\") pod \"maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm\" (UID: \"c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm" Nov 03 21:14:34.328486 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:34.328450 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-0f8726-serving-cert\"" Nov 03 21:14:34.333138 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:34.333105 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm"] Nov 03 21:14:34.411289 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:34.411253 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw9qh\" (UniqueName: \"kubernetes.io/projected/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-kube-api-access-xw9qh\") pod \"maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm\" (UID: \"c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm" Nov 03 21:14:34.411492 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:34.411398 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm\" (UID: \"c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm" Nov 03 21:14:34.411492 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:34.411461 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm\" (UID: \"c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm" Nov 03 21:14:34.411603 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:34.411504 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-metrics-certs\") pod \"maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm\" (UID: \"c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm" Nov 03 21:14:34.411756 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:14:34.411733 5615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-serving-cert: secret "maas-api-e2e-ipp-a-0f8726-serving-cert" not found Nov 03 21:14:34.411819 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:14:34.411805 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-maas-api-tls podName:c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:34.911785132 +0000 UTC m=+1569.379563852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-maas-api-tls") pod "maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm" (UID: "c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867") : secret "maas-api-e2e-ipp-a-0f8726-serving-cert" not found Nov 03 21:14:34.413158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:34.413116 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm\" (UID: \"c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm" Nov 03 21:14:34.415398 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:34.415373 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-metrics-certs\") pod \"maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm\" (UID: \"c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm" Nov 03 21:14:34.472188 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:34.472149 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw9qh\" (UniqueName: \"kubernetes.io/projected/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-kube-api-access-xw9qh\") pod \"maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm\" (UID: \"c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm" Nov 03 21:14:34.933071 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:34.933038 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm\" (UID: \"c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm" Nov 03 21:14:34.935700 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:34.935677 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm\" (UID: \"c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm" Nov 03 21:14:35.208148 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:35.203972 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm" Nov 03 21:14:35.783134 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:35.776398 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm"] Nov 03 21:14:36.271992 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:36.271947 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm" event={"ID":"c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867","Type":"ContainerStarted","Data":"57a74129ae2a6b02f1a96503a645811784301f1e597b55ffde8546a646b6ddce"} Nov 03 21:14:37.279387 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:37.279347 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm" event={"ID":"c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867","Type":"ContainerStarted","Data":"5c6f5dcac0507028db1510fd254d52ff4022dc6bc988b077f5fa3419bbd90d31"} Nov 03 21:14:37.279808 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:37.279431 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm" Nov 03 21:14:37.405095 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:37.396852 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm" podStartSLOduration=2.4845583749999998 podStartE2EDuration="3.39683244s" totalImagesPullingTime="912.274065ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:14:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:37.392522276 +0000 UTC m=+1571.860301001" watchObservedRunningTime="2026-11-03 21:14:37.39683244 +0000 UTC m=+1571.864611164" Nov 03 21:14:40.620193 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:14:40.620155 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:14:42.576935 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:42.573237 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-gtbjq"] Nov 03 21:14:42.579328 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:42.579299 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-gtbjq" Nov 03 21:14:42.629505 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:42.629461 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-gtbjq"] Nov 03 21:14:42.649691 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:42.649647 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-32f6e43c\"" Nov 03 21:14:42.673970 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:42.673920 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf675e88-d437-4eba-8bb5-d602bee8133f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-gtbjq\" (UID: \"cf675e88-d437-4eba-8bb5-d602bee8133f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-gtbjq" Nov 03 21:14:42.674400 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:42.674374 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t24vf\" (UniqueName: \"kubernetes.io/projected/cf675e88-d437-4eba-8bb5-d602bee8133f-kube-api-access-t24vf\") pod \"payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-gtbjq\" (UID: \"cf675e88-d437-4eba-8bb5-d602bee8133f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-gtbjq" Nov 03 21:14:42.813096 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:42.808177 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t24vf\" (UniqueName: \"kubernetes.io/projected/cf675e88-d437-4eba-8bb5-d602bee8133f-kube-api-access-t24vf\") pod \"payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-gtbjq\" (UID: \"cf675e88-d437-4eba-8bb5-d602bee8133f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-gtbjq" Nov 03 21:14:42.813096 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:42.808343 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf675e88-d437-4eba-8bb5-d602bee8133f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-gtbjq\" (UID: \"cf675e88-d437-4eba-8bb5-d602bee8133f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-gtbjq" Nov 03 21:14:42.813096 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:42.809303 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf675e88-d437-4eba-8bb5-d602bee8133f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-gtbjq\" (UID: \"cf675e88-d437-4eba-8bb5-d602bee8133f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-gtbjq" Nov 03 21:14:42.931166 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:42.918938 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t24vf\" (UniqueName: \"kubernetes.io/projected/cf675e88-d437-4eba-8bb5-d602bee8133f-kube-api-access-t24vf\") pod \"payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-gtbjq\" (UID: \"cf675e88-d437-4eba-8bb5-d602bee8133f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-gtbjq" Nov 03 21:14:42.948427 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:42.948389 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-gtbjq" Nov 03 21:14:43.291827 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:43.291796 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm" Nov 03 21:14:43.383844 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:43.383804 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-gtbjq"] Nov 03 21:14:43.385629 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:14:43.385598 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf675e88_d437_4eba_8bb5_d602bee8133f.slice/crio-7671c450505b6fb667a1dab822457d9c10a400972ae5c8598aa3c83b04622512 WatchSource:0}: Error finding container 7671c450505b6fb667a1dab822457d9c10a400972ae5c8598aa3c83b04622512: Status 404 returned error can't find the container with id 7671c450505b6fb667a1dab822457d9c10a400972ae5c8598aa3c83b04622512 Nov 03 21:14:44.401890 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:44.401841 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-gtbjq" event={"ID":"cf675e88-d437-4eba-8bb5-d602bee8133f","Type":"ContainerStarted","Data":"390feca44a4873bb90afd01116e9d8bb6285dfe351bd866b80ad60e39c4df55a"} Nov 03 21:14:44.402510 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:44.401919 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-gtbjq" event={"ID":"cf675e88-d437-4eba-8bb5-d602bee8133f","Type":"ContainerStarted","Data":"7671c450505b6fb667a1dab822457d9c10a400972ae5c8598aa3c83b04622512"} Nov 03 21:14:44.402604 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:44.402578 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-gtbjq" Nov 03 21:14:44.514963 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:44.514906 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-gtbjq" podStartSLOduration=2.514890252 podStartE2EDuration="2.514890252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:44.512539552 +0000 UTC m=+1578.980318278" watchObservedRunningTime="2026-11-03 21:14:44.514890252 +0000 UTC m=+1578.982668975" Nov 03 21:14:50.120348 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:50.118920 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2"] Nov 03 21:14:50.120348 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:50.119252 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" podUID="a388fecb-f964-46d3-a07e-4a87bf4c8981" containerName="istio-proxy" containerID="cri-o://fe3c5404b12c1f35774178afcfc12cb47a373e22f4311a68e145fc8f98e171b7" gracePeriod=30 Nov 03 21:14:51.241603 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:51.241563 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv"] Nov 03 21:14:51.242096 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:51.241785 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" podUID="e81a25cc-db37-460c-8adb-9cfd35690280" containerName="istio-proxy" containerID="cri-o://0f698d270f80b858eea9d495c84f186e363e56702146cfe3d1d3d8535a1bfd22" gracePeriod=30 Nov 03 21:14:51.288585 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:51.288550 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp"] Nov 03 21:14:51.288783 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:51.288763 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" podUID="c7b4be3a-5908-4a2f-89b4-54308db81b2d" containerName="istio-proxy" containerID="cri-o://30ba6f47b14fef29588ded4d27eb35236a0c6644e9e69e453291789680878bcd" gracePeriod=30 Nov 03 21:14:51.291529 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:51.291502 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc"] Nov 03 21:14:51.291711 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:51.291690 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" podUID="6a9c6e9e-a20e-40e4-a4a1-207a92d7375f" containerName="istio-proxy" containerID="cri-o://28d10fc2e4fde65a1406a814a096bc131cfa2fd91d26109f5756b86de88e8602" gracePeriod=30 Nov 03 21:14:52.806188 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:52.806153 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh"] Nov 03 21:14:52.811150 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:52.811124 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:52.887084 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:52.887027 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh"] Nov 03 21:14:52.900909 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:52.900867 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-istio-envoy\") pod \"e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:52.901136 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:52.900931 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-workload-certs\") pod \"e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:52.901136 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:52.901117 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-credential-socket\") pod \"e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:52.901259 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:52.901152 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-istio-data\") pod \"e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:52.901259 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:52.901184 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f4da92-abee-4727-90cc-e9f03dadaa12-istio-podinfo\") pod \"e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:52.901365 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:52.901306 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f4da92-abee-4727-90cc-e9f03dadaa12-istio-token\") pod \"e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:52.901365 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:52.901338 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwj6h\" (UniqueName: \"kubernetes.io/projected/26f4da92-abee-4727-90cc-e9f03dadaa12-kube-api-access-dwj6h\") pod \"e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:52.901456 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:52.901437 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-workload-socket\") pod \"e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:52.901536 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:52.901518 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f4da92-abee-4727-90cc-e9f03dadaa12-istiod-ca-cert\") pod \"e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:52.908449 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:52.908421 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-812243-openshift-default-dockercfg-5jb5b\"" Nov 03 21:14:53.006095 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:53.005888 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f4da92-abee-4727-90cc-e9f03dadaa12-istiod-ca-cert\") pod \"e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:53.006095 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:53.005068 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f4da92-abee-4727-90cc-e9f03dadaa12-istiod-ca-cert\") pod \"e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:53.006095 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:53.006025 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-istio-envoy\") pod \"e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:53.006095 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:53.006093 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-workload-certs\") pod \"e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:53.006471 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:53.006245 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-credential-socket\") pod \"e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:53.006471 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:53.006277 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-istio-data\") pod \"e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:53.006471 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:53.006310 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f4da92-abee-4727-90cc-e9f03dadaa12-istio-podinfo\") pod \"e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:53.006471 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:53.006397 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f4da92-abee-4727-90cc-e9f03dadaa12-istio-token\") pod \"e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:53.006471 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:53.006421 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwj6h\" (UniqueName: \"kubernetes.io/projected/26f4da92-abee-4727-90cc-e9f03dadaa12-kube-api-access-dwj6h\") pod \"e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:53.006730 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:53.006503 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-workload-socket\") pod \"e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:53.011100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:53.006891 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-workload-socket\") pod \"e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:53.020175 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:53.020137 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-workload-certs\") pod \"e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:53.022373 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:53.022280 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-credential-socket\") pod \"e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:53.023339 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:53.023309 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-istio-data\") pod \"e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:53.058162 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:53.036690 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f4da92-abee-4727-90cc-e9f03dadaa12-istio-podinfo\") pod \"e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:53.058162 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:53.037340 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-istio-envoy\") pod \"e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:53.085999 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:53.085966 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f4da92-abee-4727-90cc-e9f03dadaa12-istio-token\") pod \"e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:53.090149 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:53.090034 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwj6h\" (UniqueName: \"kubernetes.io/projected/26f4da92-abee-4727-90cc-e9f03dadaa12-kube-api-access-dwj6h\") pod \"e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:53.131449 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:53.131406 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:53.635376 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:53.635345 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh"] Nov 03 21:14:53.636350 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:14:53.636319 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f4da92_abee_4727_90cc_e9f03dadaa12.slice/crio-f9af99ea3c12f0e7a0b1eb1a966e92d5a167d357d463cc0a261d504164b9f0e8 WatchSource:0}: Error finding container f9af99ea3c12f0e7a0b1eb1a966e92d5a167d357d463cc0a261d504164b9f0e8: Status 404 returned error can't find the container with id f9af99ea3c12f0e7a0b1eb1a966e92d5a167d357d463cc0a261d504164b9f0e8 Nov 03 21:14:53.639163 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:53.639112 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:14:53.639257 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:53.639198 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:14:53.639257 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:53.639230 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:14:54.545521 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:54.545481 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" event={"ID":"26f4da92-abee-4727-90cc-e9f03dadaa12","Type":"ContainerStarted","Data":"3cf67f2f13c4a378830b88a229fe9a0d80b115e78e470e22613e19874b4df0fc"} Nov 03 21:14:54.545910 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:54.545574 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" event={"ID":"26f4da92-abee-4727-90cc-e9f03dadaa12","Type":"ContainerStarted","Data":"f9af99ea3c12f0e7a0b1eb1a966e92d5a167d357d463cc0a261d504164b9f0e8"} Nov 03 21:14:54.756350 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:54.756294 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" podStartSLOduration=2.756279858 podStartE2EDuration="2.756279858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:54.706492381 +0000 UTC m=+1589.174271105" watchObservedRunningTime="2026-11-03 21:14:54.756279858 +0000 UTC m=+1589.224058577" Nov 03 21:14:55.035001 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.034962 5615 patch_prober.go:29] interesting pod/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:14:55.035218 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.035042 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" podUID="a388fecb-f964-46d3-a07e-4a87bf4c8981" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:14:55.132847 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.132788 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:55.138991 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.138958 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:55.395413 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.395385 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:14:55.411748 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.411718 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-gtbjq" Nov 03 21:14:55.446583 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.446553 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a388fecb-f964-46d3-a07e-4a87bf4c8981-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a388fecb-f964-46d3-a07e-4a87bf4c8981-istiod-ca-cert\") pod \"a388fecb-f964-46d3-a07e-4a87bf4c8981\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " Nov 03 21:14:55.446762 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.446618 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-credential-socket\") pod \"a388fecb-f964-46d3-a07e-4a87bf4c8981\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " Nov 03 21:14:55.446762 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.446665 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a388fecb-f964-46d3-a07e-4a87bf4c8981-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a388fecb-f964-46d3-a07e-4a87bf4c8981-istio-podinfo\") pod \"a388fecb-f964-46d3-a07e-4a87bf4c8981\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " Nov 03 21:14:55.446762 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.446732 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a388fecb-f964-46d3-a07e-4a87bf4c8981-istio-token\" (UniqueName: \"kubernetes.io/projected/a388fecb-f964-46d3-a07e-4a87bf4c8981-istio-token\") pod \"a388fecb-f964-46d3-a07e-4a87bf4c8981\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " Nov 03 21:14:55.446952 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.446775 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a388fecb-f964-46d3-a07e-4a87bf4c8981-kube-api-access-77nfc\" (UniqueName: \"kubernetes.io/projected/a388fecb-f964-46d3-a07e-4a87bf4c8981-kube-api-access-77nfc\") pod \"a388fecb-f964-46d3-a07e-4a87bf4c8981\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " Nov 03 21:14:55.446952 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.446810 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-istio-data\") pod \"a388fecb-f964-46d3-a07e-4a87bf4c8981\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " Nov 03 21:14:55.446952 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.446870 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-workload-socket\") pod \"a388fecb-f964-46d3-a07e-4a87bf4c8981\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " Nov 03 21:14:55.446952 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.446948 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-workload-certs\") pod \"a388fecb-f964-46d3-a07e-4a87bf4c8981\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " Nov 03 21:14:55.447238 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.446983 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-credential-socket" pod "a388fecb-f964-46d3-a07e-4a87bf4c8981" (UID: "a388fecb-f964-46d3-a07e-4a87bf4c8981"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:55.447238 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.447020 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a388fecb-f964-46d3-a07e-4a87bf4c8981-istiod-ca-cert" pod "a388fecb-f964-46d3-a07e-4a87bf4c8981" (UID: "a388fecb-f964-46d3-a07e-4a87bf4c8981"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:55.447238 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.447027 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-istio-envoy\") pod \"a388fecb-f964-46d3-a07e-4a87bf4c8981\" (UID: \"a388fecb-f964-46d3-a07e-4a87bf4c8981\") " Nov 03 21:14:55.447389 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.447346 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-workload-certs" pod "a388fecb-f964-46d3-a07e-4a87bf4c8981" (UID: "a388fecb-f964-46d3-a07e-4a87bf4c8981"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:55.447389 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.447371 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-workload-socket" pod "a388fecb-f964-46d3-a07e-4a87bf4c8981" (UID: "a388fecb-f964-46d3-a07e-4a87bf4c8981"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:55.447504 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.447473 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-istio-data" pod "a388fecb-f964-46d3-a07e-4a87bf4c8981" (UID: "a388fecb-f964-46d3-a07e-4a87bf4c8981"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:55.447984 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.447957 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:55.448165 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.447987 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:55.448165 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.448004 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:55.448165 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.448019 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a388fecb-f964-46d3-a07e-4a87bf4c8981-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:55.448165 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.448034 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:55.449302 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.449281 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a388fecb-f964-46d3-a07e-4a87bf4c8981-istio-podinfo" pod "a388fecb-f964-46d3-a07e-4a87bf4c8981" (UID: "a388fecb-f964-46d3-a07e-4a87bf4c8981"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:14:55.449389 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.449298 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a388fecb-f964-46d3-a07e-4a87bf4c8981-istio-token" pod "a388fecb-f964-46d3-a07e-4a87bf4c8981" (UID: "a388fecb-f964-46d3-a07e-4a87bf4c8981"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:55.449567 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.449535 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a388fecb-f964-46d3-a07e-4a87bf4c8981-kube-api-access-77nfc" pod "a388fecb-f964-46d3-a07e-4a87bf4c8981" (UID: "a388fecb-f964-46d3-a07e-4a87bf4c8981"). InnerVolumeSpecName "kube-api-access-77nfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:55.449567 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.449538 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-istio-envoy" pod "a388fecb-f964-46d3-a07e-4a87bf4c8981" (UID: "a388fecb-f964-46d3-a07e-4a87bf4c8981"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:55.552151 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.551845 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77nfc\" (UniqueName: \"kubernetes.io/projected/a388fecb-f964-46d3-a07e-4a87bf4c8981-kube-api-access-77nfc\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:55.552151 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.551885 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a388fecb-f964-46d3-a07e-4a87bf4c8981-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:55.552151 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.551902 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a388fecb-f964-46d3-a07e-4a87bf4c8981-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:55.552151 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.551915 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a388fecb-f964-46d3-a07e-4a87bf4c8981-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:55.606321 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.605393 5615 generic.go:356] "Generic (PLEG): container finished" podID="a388fecb-f964-46d3-a07e-4a87bf4c8981" containerID="fe3c5404b12c1f35774178afcfc12cb47a373e22f4311a68e145fc8f98e171b7" exitCode=0 Nov 03 21:14:55.606321 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.606211 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" event={"ID":"a388fecb-f964-46d3-a07e-4a87bf4c8981","Type":"ContainerDied","Data":"fe3c5404b12c1f35774178afcfc12cb47a373e22f4311a68e145fc8f98e171b7"} Nov 03 21:14:55.606321 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.606230 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" Nov 03 21:14:55.606321 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.606265 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2" event={"ID":"a388fecb-f964-46d3-a07e-4a87bf4c8981","Type":"ContainerDied","Data":"28dc8ba9d6b188f18d285195987f6f60d11da60503d78d0b598166ba202ca79d"} Nov 03 21:14:55.606321 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.606280 5615 scope.go:122] "RemoveContainer" containerID="fe3c5404b12c1f35774178afcfc12cb47a373e22f4311a68e145fc8f98e171b7" Nov 03 21:14:55.606719 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.606708 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:55.608631 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.608574 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:14:55.620243 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.620160 5615 scope.go:122] "RemoveContainer" containerID="1cfb6ca4f7f451d6f08c6e8d975894c03497943625786f17f4c97add819261b8" Nov 03 21:14:55.630594 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.630567 5615 scope.go:122] "RemoveContainer" containerID="fe3c5404b12c1f35774178afcfc12cb47a373e22f4311a68e145fc8f98e171b7" Nov 03 21:14:55.630901 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:14:55.630873 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe3c5404b12c1f35774178afcfc12cb47a373e22f4311a68e145fc8f98e171b7\": container with ID starting with fe3c5404b12c1f35774178afcfc12cb47a373e22f4311a68e145fc8f98e171b7 not found: ID does not exist" containerID="fe3c5404b12c1f35774178afcfc12cb47a373e22f4311a68e145fc8f98e171b7" Nov 03 21:14:55.630969 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.630918 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe3c5404b12c1f35774178afcfc12cb47a373e22f4311a68e145fc8f98e171b7"} err="failed to get container status \"fe3c5404b12c1f35774178afcfc12cb47a373e22f4311a68e145fc8f98e171b7\": rpc error: code = NotFound desc = could not find container \"fe3c5404b12c1f35774178afcfc12cb47a373e22f4311a68e145fc8f98e171b7\": container with ID starting with fe3c5404b12c1f35774178afcfc12cb47a373e22f4311a68e145fc8f98e171b7 not found: ID does not exist" Nov 03 21:14:55.630969 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.630938 5615 scope.go:122] "RemoveContainer" containerID="1cfb6ca4f7f451d6f08c6e8d975894c03497943625786f17f4c97add819261b8" Nov 03 21:14:55.631334 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:14:55.631310 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cfb6ca4f7f451d6f08c6e8d975894c03497943625786f17f4c97add819261b8\": container with ID starting with 1cfb6ca4f7f451d6f08c6e8d975894c03497943625786f17f4c97add819261b8 not found: ID does not exist" containerID="1cfb6ca4f7f451d6f08c6e8d975894c03497943625786f17f4c97add819261b8" Nov 03 21:14:55.631424 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.631338 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cfb6ca4f7f451d6f08c6e8d975894c03497943625786f17f4c97add819261b8"} err="failed to get container status \"1cfb6ca4f7f451d6f08c6e8d975894c03497943625786f17f4c97add819261b8\": rpc error: code = NotFound desc = could not find container \"1cfb6ca4f7f451d6f08c6e8d975894c03497943625786f17f4c97add819261b8\": container with ID starting with 1cfb6ca4f7f451d6f08c6e8d975894c03497943625786f17f4c97add819261b8 not found: ID does not exist" Nov 03 21:14:55.761439 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.761400 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2"] Nov 03 21:14:55.778113 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:55.778059 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-6ad223-openshift-default-784cf587bc-vr5w2"] Nov 03 21:14:56.587565 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.587536 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:14:56.617173 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.617136 5615 generic.go:356] "Generic (PLEG): container finished" podID="e81a25cc-db37-460c-8adb-9cfd35690280" containerID="0f698d270f80b858eea9d495c84f186e363e56702146cfe3d1d3d8535a1bfd22" exitCode=0 Nov 03 21:14:56.617302 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.617196 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" Nov 03 21:14:56.617302 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.617191 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" event={"ID":"e81a25cc-db37-460c-8adb-9cfd35690280","Type":"ContainerDied","Data":"0f698d270f80b858eea9d495c84f186e363e56702146cfe3d1d3d8535a1bfd22"} Nov 03 21:14:56.617454 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.617326 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv" event={"ID":"e81a25cc-db37-460c-8adb-9cfd35690280","Type":"ContainerDied","Data":"ef45555f97030ea557a8b9707921ab23f82f806e9a1dfec6c800033a471aedb9"} Nov 03 21:14:56.617454 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.617360 5615 scope.go:122] "RemoveContainer" containerID="0f698d270f80b858eea9d495c84f186e363e56702146cfe3d1d3d8535a1bfd22" Nov 03 21:14:56.619178 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.619152 5615 generic.go:356] "Generic (PLEG): container finished" podID="c7b4be3a-5908-4a2f-89b4-54308db81b2d" containerID="30ba6f47b14fef29588ded4d27eb35236a0c6644e9e69e453291789680878bcd" exitCode=0 Nov 03 21:14:56.619292 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.619249 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" event={"ID":"c7b4be3a-5908-4a2f-89b4-54308db81b2d","Type":"ContainerDied","Data":"30ba6f47b14fef29588ded4d27eb35236a0c6644e9e69e453291789680878bcd"} Nov 03 21:14:56.626255 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.626233 5615 generic.go:356] "Generic (PLEG): container finished" podID="6a9c6e9e-a20e-40e4-a4a1-207a92d7375f" containerID="28d10fc2e4fde65a1406a814a096bc131cfa2fd91d26109f5756b86de88e8602" exitCode=0 Nov 03 21:14:56.626378 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.626311 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" event={"ID":"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f","Type":"ContainerDied","Data":"28d10fc2e4fde65a1406a814a096bc131cfa2fd91d26109f5756b86de88e8602"} Nov 03 21:14:56.626378 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.626361 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" event={"ID":"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f","Type":"ContainerDied","Data":"efa46e3eb9a2fead4f23cc031d29bdf59e91947d9b80e997862ff6b5a27475f0"} Nov 03 21:14:56.626510 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.626385 5615 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efa46e3eb9a2fead4f23cc031d29bdf59e91947d9b80e997862ff6b5a27475f0" Nov 03 21:14:56.628990 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.628969 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:14:56.630368 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.630342 5615 scope.go:122] "RemoveContainer" containerID="0f698d270f80b858eea9d495c84f186e363e56702146cfe3d1d3d8535a1bfd22" Nov 03 21:14:56.630642 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:14:56.630623 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f698d270f80b858eea9d495c84f186e363e56702146cfe3d1d3d8535a1bfd22\": container with ID starting with 0f698d270f80b858eea9d495c84f186e363e56702146cfe3d1d3d8535a1bfd22 not found: ID does not exist" containerID="0f698d270f80b858eea9d495c84f186e363e56702146cfe3d1d3d8535a1bfd22" Nov 03 21:14:56.630708 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.630649 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f698d270f80b858eea9d495c84f186e363e56702146cfe3d1d3d8535a1bfd22"} err="failed to get container status \"0f698d270f80b858eea9d495c84f186e363e56702146cfe3d1d3d8535a1bfd22\": rpc error: code = NotFound desc = could not find container \"0f698d270f80b858eea9d495c84f186e363e56702146cfe3d1d3d8535a1bfd22\": container with ID starting with 0f698d270f80b858eea9d495c84f186e363e56702146cfe3d1d3d8535a1bfd22 not found: ID does not exist" Nov 03 21:14:56.646548 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.646526 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:14:56.679782 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.679740 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e81a25cc-db37-460c-8adb-9cfd35690280-kube-api-access-drb25\" (UniqueName: \"kubernetes.io/projected/e81a25cc-db37-460c-8adb-9cfd35690280-kube-api-access-drb25\") pod \"e81a25cc-db37-460c-8adb-9cfd35690280\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " Nov 03 21:14:56.679933 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.679789 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-workload-socket\") pod \"e81a25cc-db37-460c-8adb-9cfd35690280\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " Nov 03 21:14:56.679933 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.679842 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e81a25cc-db37-460c-8adb-9cfd35690280-istio-token\" (UniqueName: \"kubernetes.io/projected/e81a25cc-db37-460c-8adb-9cfd35690280-istio-token\") pod \"e81a25cc-db37-460c-8adb-9cfd35690280\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " Nov 03 21:14:56.680043 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.679936 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-workload-socket\") pod \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " Nov 03 21:14:56.680151 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.680062 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istio-data\") pod \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " Nov 03 21:14:56.680151 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.680132 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-workload-socket" pod "e81a25cc-db37-460c-8adb-9cfd35690280" (UID: "e81a25cc-db37-460c-8adb-9cfd35690280"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:56.680272 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.680139 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-kube-api-access-pjlm6\" (UniqueName: \"kubernetes.io/projected/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-kube-api-access-pjlm6\") pod \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " Nov 03 21:14:56.680272 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.680245 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-workload-socket" pod "c7b4be3a-5908-4a2f-89b4-54308db81b2d" (UID: "c7b4be3a-5908-4a2f-89b4-54308db81b2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:56.680272 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.680257 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istio-token\" (UniqueName: \"kubernetes.io/projected/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istio-token\") pod \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " Nov 03 21:14:56.680435 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.680300 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istio-data" pod "6a9c6e9e-a20e-40e4-a4a1-207a92d7375f" (UID: "6a9c6e9e-a20e-40e4-a4a1-207a92d7375f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:56.680435 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.680337 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-workload-certs\") pod \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " Nov 03 21:14:56.680435 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.680372 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e81a25cc-db37-460c-8adb-9cfd35690280-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e81a25cc-db37-460c-8adb-9cfd35690280-istio-podinfo\") pod \"e81a25cc-db37-460c-8adb-9cfd35690280\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " Nov 03 21:14:56.680435 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.680422 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istio-data\") pod \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " Nov 03 21:14:56.680652 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.680478 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istio-podinfo\") pod \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " Nov 03 21:14:56.680652 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.680513 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istio-envoy\") pod \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " Nov 03 21:14:56.680652 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.680578 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-credential-socket\") pod \"e81a25cc-db37-460c-8adb-9cfd35690280\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " Nov 03 21:14:56.680652 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.680599 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-workload-certs" pod "6a9c6e9e-a20e-40e4-a4a1-207a92d7375f" (UID: "6a9c6e9e-a20e-40e4-a4a1-207a92d7375f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:56.680652 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.680614 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-credential-socket\") pod \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " Nov 03 21:14:56.680652 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.680622 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istio-data" pod "c7b4be3a-5908-4a2f-89b4-54308db81b2d" (UID: "c7b4be3a-5908-4a2f-89b4-54308db81b2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:56.680652 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.680652 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istio-token\" (UniqueName: \"kubernetes.io/projected/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istio-token\") pod \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " Nov 03 21:14:56.681000 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.680690 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istiod-ca-cert\") pod \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " Nov 03 21:14:56.681000 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.680729 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-credential-socket\") pod \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " Nov 03 21:14:56.681000 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.680764 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istiod-ca-cert\") pod \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " Nov 03 21:14:56.681000 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.680793 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-credential-socket" pod "e81a25cc-db37-460c-8adb-9cfd35690280" (UID: "e81a25cc-db37-460c-8adb-9cfd35690280"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:56.681000 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.680829 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-workload-certs\") pod \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " Nov 03 21:14:56.681000 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.680867 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-istio-envoy\") pod \"e81a25cc-db37-460c-8adb-9cfd35690280\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " Nov 03 21:14:56.681000 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.680959 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-istio-data\") pod \"e81a25cc-db37-460c-8adb-9cfd35690280\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " Nov 03 21:14:56.681000 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.680996 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b4be3a-5908-4a2f-89b4-54308db81b2d-kube-api-access-b5n7n\" (UniqueName: \"kubernetes.io/projected/c7b4be3a-5908-4a2f-89b4-54308db81b2d-kube-api-access-b5n7n\") pod \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " Nov 03 21:14:56.681445 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.681063 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istio-envoy\") pod \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " Nov 03 21:14:56.681445 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.681128 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-workload-socket\") pod \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\" (UID: \"6a9c6e9e-a20e-40e4-a4a1-207a92d7375f\") " Nov 03 21:14:56.681445 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.681170 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istio-podinfo\") pod \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\" (UID: \"c7b4be3a-5908-4a2f-89b4-54308db81b2d\") " Nov 03 21:14:56.681445 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.681206 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-workload-certs\") pod \"e81a25cc-db37-460c-8adb-9cfd35690280\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " Nov 03 21:14:56.681445 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.681243 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e81a25cc-db37-460c-8adb-9cfd35690280-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e81a25cc-db37-460c-8adb-9cfd35690280-istiod-ca-cert\") pod \"e81a25cc-db37-460c-8adb-9cfd35690280\" (UID: \"e81a25cc-db37-460c-8adb-9cfd35690280\") " Nov 03 21:14:56.682179 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.682155 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:56.682326 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.682183 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:56.682326 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.682200 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:56.682326 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.682220 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:56.682326 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.682233 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:56.682326 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.682248 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:56.682768 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.682714 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istio-token" pod "c7b4be3a-5908-4a2f-89b4-54308db81b2d" (UID: "c7b4be3a-5908-4a2f-89b4-54308db81b2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:56.682768 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.682757 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-kube-api-access-pjlm6" pod "6a9c6e9e-a20e-40e4-a4a1-207a92d7375f" (UID: "6a9c6e9e-a20e-40e4-a4a1-207a92d7375f"). InnerVolumeSpecName "kube-api-access-pjlm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:56.682939 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.682782 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e81a25cc-db37-460c-8adb-9cfd35690280-istio-token" pod "e81a25cc-db37-460c-8adb-9cfd35690280" (UID: "e81a25cc-db37-460c-8adb-9cfd35690280"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:56.682939 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.682886 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-credential-socket" pod "c7b4be3a-5908-4a2f-89b4-54308db81b2d" (UID: "c7b4be3a-5908-4a2f-89b4-54308db81b2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:56.682939 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.682915 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istiod-ca-cert" pod "c7b4be3a-5908-4a2f-89b4-54308db81b2d" (UID: "c7b4be3a-5908-4a2f-89b4-54308db81b2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:56.683124 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.683009 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e81a25cc-db37-460c-8adb-9cfd35690280-istio-podinfo" pod "e81a25cc-db37-460c-8adb-9cfd35690280" (UID: "e81a25cc-db37-460c-8adb-9cfd35690280"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:14:56.683285 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.683261 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e81a25cc-db37-460c-8adb-9cfd35690280-istiod-ca-cert" pod "e81a25cc-db37-460c-8adb-9cfd35690280" (UID: "e81a25cc-db37-460c-8adb-9cfd35690280"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:56.683357 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.683338 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e81a25cc-db37-460c-8adb-9cfd35690280-kube-api-access-drb25" pod "e81a25cc-db37-460c-8adb-9cfd35690280" (UID: "e81a25cc-db37-460c-8adb-9cfd35690280"). InnerVolumeSpecName "kube-api-access-drb25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:56.683779 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.683521 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-workload-certs" pod "e81a25cc-db37-460c-8adb-9cfd35690280" (UID: "e81a25cc-db37-460c-8adb-9cfd35690280"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:56.683779 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.683681 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-workload-certs" pod "c7b4be3a-5908-4a2f-89b4-54308db81b2d" (UID: "c7b4be3a-5908-4a2f-89b4-54308db81b2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:56.683779 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.683737 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-workload-socket" pod "6a9c6e9e-a20e-40e4-a4a1-207a92d7375f" (UID: "6a9c6e9e-a20e-40e4-a4a1-207a92d7375f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:56.683998 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.683834 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-istio-data" pod "e81a25cc-db37-460c-8adb-9cfd35690280" (UID: "e81a25cc-db37-460c-8adb-9cfd35690280"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:56.683998 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.683907 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-credential-socket" pod "6a9c6e9e-a20e-40e4-a4a1-207a92d7375f" (UID: "6a9c6e9e-a20e-40e4-a4a1-207a92d7375f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:56.684141 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.684062 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istiod-ca-cert" pod "6a9c6e9e-a20e-40e4-a4a1-207a92d7375f" (UID: "6a9c6e9e-a20e-40e4-a4a1-207a92d7375f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:56.684418 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.684391 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istio-envoy" pod "c7b4be3a-5908-4a2f-89b4-54308db81b2d" (UID: "c7b4be3a-5908-4a2f-89b4-54308db81b2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:56.684701 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.684668 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istio-token" pod "6a9c6e9e-a20e-40e4-a4a1-207a92d7375f" (UID: "6a9c6e9e-a20e-40e4-a4a1-207a92d7375f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:56.685049 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.685020 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istio-envoy" pod "6a9c6e9e-a20e-40e4-a4a1-207a92d7375f" (UID: "6a9c6e9e-a20e-40e4-a4a1-207a92d7375f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:56.685338 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.685319 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istio-podinfo" pod "6a9c6e9e-a20e-40e4-a4a1-207a92d7375f" (UID: "6a9c6e9e-a20e-40e4-a4a1-207a92d7375f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:14:56.685389 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.685367 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istio-podinfo" pod "c7b4be3a-5908-4a2f-89b4-54308db81b2d" (UID: "c7b4be3a-5908-4a2f-89b4-54308db81b2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:14:56.685738 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.685715 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-istio-envoy" pod "e81a25cc-db37-460c-8adb-9cfd35690280" (UID: "e81a25cc-db37-460c-8adb-9cfd35690280"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:56.686045 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.686030 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b4be3a-5908-4a2f-89b4-54308db81b2d-kube-api-access-b5n7n" pod "c7b4be3a-5908-4a2f-89b4-54308db81b2d" (UID: "c7b4be3a-5908-4a2f-89b4-54308db81b2d"). InnerVolumeSpecName "kube-api-access-b5n7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:56.783939 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.783895 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:56.783939 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.783933 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5n7n\" (UniqueName: \"kubernetes.io/projected/c7b4be3a-5908-4a2f-89b4-54308db81b2d-kube-api-access-b5n7n\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:56.783939 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.783945 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:56.784243 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.783955 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:56.784243 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.783964 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:56.784243 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.783973 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:56.784243 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.783981 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e81a25cc-db37-460c-8adb-9cfd35690280-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:56.784243 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.783991 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drb25\" (UniqueName: \"kubernetes.io/projected/e81a25cc-db37-460c-8adb-9cfd35690280-kube-api-access-drb25\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:56.784243 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.783999 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e81a25cc-db37-460c-8adb-9cfd35690280-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:56.784243 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.784008 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjlm6\" (UniqueName: \"kubernetes.io/projected/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-kube-api-access-pjlm6\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:56.784243 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.784016 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:56.784243 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.784024 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e81a25cc-db37-460c-8adb-9cfd35690280-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:56.784243 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.784033 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:56.784243 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.784041 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:56.784243 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.784049 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:56.784243 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.784057 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:56.784243 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.784064 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b4be3a-5908-4a2f-89b4-54308db81b2d-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:56.784243 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.784085 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:56.784243 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.784094 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:56.784243 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.784102 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b4be3a-5908-4a2f-89b4-54308db81b2d-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:56.784243 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:56.784110 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e81a25cc-db37-460c-8adb-9cfd35690280-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:57.121154 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:57.118710 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv"] Nov 03 21:14:57.123425 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:57.123393 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-wj5hv"] Nov 03 21:14:57.638863 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:57.638783 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" Nov 03 21:14:57.638863 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:57.638815 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc" Nov 03 21:14:57.638863 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:57.638809 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp" event={"ID":"c7b4be3a-5908-4a2f-89b4-54308db81b2d","Type":"ContainerDied","Data":"7578bee8bccc123ffe84b11963038db0862572b9f063b6e253cf15ebe62eef7c"} Nov 03 21:14:57.639457 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:57.638915 5615 scope.go:122] "RemoveContainer" containerID="30ba6f47b14fef29588ded4d27eb35236a0c6644e9e69e453291789680878bcd" Nov 03 21:14:57.652715 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:57.652687 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a388fecb-f964-46d3-a07e-4a87bf4c8981" path="/var/lib/kubelet/pods/a388fecb-f964-46d3-a07e-4a87bf4c8981/volumes" Nov 03 21:14:57.653272 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:57.653258 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e81a25cc-db37-460c-8adb-9cfd35690280" path="/var/lib/kubelet/pods/e81a25cc-db37-460c-8adb-9cfd35690280/volumes" Nov 03 21:14:57.825649 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:57.825612 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp"] Nov 03 21:14:57.878788 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:57.878750 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q7xdp"] Nov 03 21:14:57.983359 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:57.983326 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc"] Nov 03 21:14:57.988244 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:57.988215 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-bx4mc"] Nov 03 21:14:59.653968 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:59.653933 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a9c6e9e-a20e-40e4-a4a1-207a92d7375f" path="/var/lib/kubelet/pods/6a9c6e9e-a20e-40e4-a4a1-207a92d7375f/volumes" Nov 03 21:14:59.654475 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:14:59.654459 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7b4be3a-5908-4a2f-89b4-54308db81b2d" path="/var/lib/kubelet/pods/c7b4be3a-5908-4a2f-89b4-54308db81b2d/volumes" Nov 03 21:15:02.761098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:02.760559 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-812243-75947bbfb8-lnlqk"] Nov 03 21:15:02.790100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:02.761597 5615 cpu_manager.go:414] "containerMap: removing container" podUID="e81a25cc-db37-460c-8adb-9cfd35690280" containerName="istio-proxy" Nov 03 21:15:02.790100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:02.761621 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e81a25cc-db37-460c-8adb-9cfd35690280" containerName="istio-proxy" Nov 03 21:15:02.790100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:02.761655 5615 cpu_manager.go:414] "containerMap: removing container" podUID="c7b4be3a-5908-4a2f-89b4-54308db81b2d" containerName="istio-proxy" Nov 03 21:15:02.790100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:02.761664 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b4be3a-5908-4a2f-89b4-54308db81b2d" containerName="istio-proxy" Nov 03 21:15:02.790100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:02.761676 5615 cpu_manager.go:414] "containerMap: removing container" podUID="a388fecb-f964-46d3-a07e-4a87bf4c8981" containerName="istio-proxy" Nov 03 21:15:02.790100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:02.761683 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a388fecb-f964-46d3-a07e-4a87bf4c8981" containerName="istio-proxy" Nov 03 21:15:02.790100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:02.761698 5615 cpu_manager.go:414] "containerMap: removing container" podUID="a388fecb-f964-46d3-a07e-4a87bf4c8981" containerName="istio-proxy" Nov 03 21:15:02.790100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:02.761707 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a388fecb-f964-46d3-a07e-4a87bf4c8981" containerName="istio-proxy" Nov 03 21:15:02.790100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:02.761766 5615 cpu_manager.go:414] "containerMap: removing container" podUID="6a9c6e9e-a20e-40e4-a4a1-207a92d7375f" containerName="istio-proxy" Nov 03 21:15:02.790100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:02.761774 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a9c6e9e-a20e-40e4-a4a1-207a92d7375f" containerName="istio-proxy" Nov 03 21:15:02.790100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:02.788426 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b4be3a-5908-4a2f-89b4-54308db81b2d" containerName="istio-proxy" Nov 03 21:15:02.790100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:02.788477 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="a388fecb-f964-46d3-a07e-4a87bf4c8981" containerName="istio-proxy" Nov 03 21:15:02.790100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:02.788489 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="a388fecb-f964-46d3-a07e-4a87bf4c8981" containerName="istio-proxy" Nov 03 21:15:02.790100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:02.788526 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a9c6e9e-a20e-40e4-a4a1-207a92d7375f" containerName="istio-proxy" Nov 03 21:15:02.790100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:02.788559 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="e81a25cc-db37-460c-8adb-9cfd35690280" containerName="istio-proxy" Nov 03 21:15:02.846949 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:02.845197 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-812243-75947bbfb8-lnlqk"] Nov 03 21:15:02.846949 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:02.845303 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-75947bbfb8-lnlqk" Nov 03 21:15:02.916136 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:02.916037 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2dtt\" (UniqueName: \"kubernetes.io/projected/c06f0f6d-1bd4-4fcc-a68a-1080bda86136-kube-api-access-v2dtt\") pod \"payload-processing-e2e-ipp-a-812243-75947bbfb8-lnlqk\" (UID: \"c06f0f6d-1bd4-4fcc-a68a-1080bda86136\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-75947bbfb8-lnlqk" Nov 03 21:15:02.916286 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:02.916243 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c06f0f6d-1bd4-4fcc-a68a-1080bda86136-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-812243-75947bbfb8-lnlqk\" (UID: \"c06f0f6d-1bd4-4fcc-a68a-1080bda86136\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-75947bbfb8-lnlqk" Nov 03 21:15:02.942214 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:02.942182 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-812243\"" Nov 03 21:15:02.942390 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:02.942320 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-812243-dockercfg-lbdzt\"" Nov 03 21:15:03.017513 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:03.017471 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c06f0f6d-1bd4-4fcc-a68a-1080bda86136-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-812243-75947bbfb8-lnlqk\" (UID: \"c06f0f6d-1bd4-4fcc-a68a-1080bda86136\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-75947bbfb8-lnlqk" Nov 03 21:15:03.017715 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:03.017685 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2dtt\" (UniqueName: \"kubernetes.io/projected/c06f0f6d-1bd4-4fcc-a68a-1080bda86136-kube-api-access-v2dtt\") pod \"payload-processing-e2e-ipp-a-812243-75947bbfb8-lnlqk\" (UID: \"c06f0f6d-1bd4-4fcc-a68a-1080bda86136\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-75947bbfb8-lnlqk" Nov 03 21:15:03.018154 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:03.018134 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c06f0f6d-1bd4-4fcc-a68a-1080bda86136-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-812243-75947bbfb8-lnlqk\" (UID: \"c06f0f6d-1bd4-4fcc-a68a-1080bda86136\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-75947bbfb8-lnlqk" Nov 03 21:15:03.082105 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:03.077561 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2dtt\" (UniqueName: \"kubernetes.io/projected/c06f0f6d-1bd4-4fcc-a68a-1080bda86136-kube-api-access-v2dtt\") pod \"payload-processing-e2e-ipp-a-812243-75947bbfb8-lnlqk\" (UID: \"c06f0f6d-1bd4-4fcc-a68a-1080bda86136\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-75947bbfb8-lnlqk" Nov 03 21:15:03.168471 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:03.168371 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-75947bbfb8-lnlqk" Nov 03 21:15:03.543549 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:03.543522 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-812243-75947bbfb8-lnlqk"] Nov 03 21:15:03.544941 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:15:03.544900 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc06f0f6d_1bd4_4fcc_a68a_1080bda86136.slice/crio-73634bfa7b6d84beafc695863d30a681827efbeb4d5517de93ad6c3347910672 WatchSource:0}: Error finding container 73634bfa7b6d84beafc695863d30a681827efbeb4d5517de93ad6c3347910672: Status 404 returned error can't find the container with id 73634bfa7b6d84beafc695863d30a681827efbeb4d5517de93ad6c3347910672 Nov 03 21:15:03.797125 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:03.797012 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-75947bbfb8-lnlqk" event={"ID":"c06f0f6d-1bd4-4fcc-a68a-1080bda86136","Type":"ContainerStarted","Data":"12cda3ae01a1a57326c1d855a1d418d458446af3c542c254754de7147108cb60"} Nov 03 21:15:03.797125 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:03.797067 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-75947bbfb8-lnlqk" event={"ID":"c06f0f6d-1bd4-4fcc-a68a-1080bda86136","Type":"ContainerStarted","Data":"73634bfa7b6d84beafc695863d30a681827efbeb4d5517de93ad6c3347910672"} Nov 03 21:15:03.797125 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:03.797122 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-75947bbfb8-lnlqk" Nov 03 21:15:03.909424 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:03.909365 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-75947bbfb8-lnlqk" podStartSLOduration=1.909348893 podStartE2EDuration="1.909348893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:03.907140868 +0000 UTC m=+1598.374919590" watchObservedRunningTime="2026-11-03 21:15:03.909348893 +0000 UTC m=+1598.377127615" Nov 03 21:15:04.505467 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:04.505434 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv"] Nov 03 21:15:04.505676 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:04.505651 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv" podUID="51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91" containerName="maas-api" containerID="cri-o://aa65af3220b554e1e7fef2e8e8f2a7980f29280f301546d4e5c7b937f211c644" gracePeriod=30 Nov 03 21:15:04.597402 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:15:04.597364 5615 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b02bb8_1b5a_4be2_92f5_8a4e3b50ef91.slice/crio-conmon-aa65af3220b554e1e7fef2e8e8f2a7980f29280f301546d4e5c7b937f211c644.scope\": RecentStats: unable to find data in memory cache]" Nov 03 21:15:04.845989 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:04.845904 5615 generic.go:356] "Generic (PLEG): container finished" podID="51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91" containerID="aa65af3220b554e1e7fef2e8e8f2a7980f29280f301546d4e5c7b937f211c644" exitCode=0 Nov 03 21:15:04.846448 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:04.845990 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv" event={"ID":"51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91","Type":"ContainerDied","Data":"aa65af3220b554e1e7fef2e8e8f2a7980f29280f301546d4e5c7b937f211c644"} Nov 03 21:15:04.880050 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:04.880023 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv" Nov 03 21:15:04.951643 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:04.951609 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-kube-api-access-wr8q4\" (UniqueName: \"kubernetes.io/projected/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-kube-api-access-wr8q4\") pod \"51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91\" (UID: \"51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91\") " Nov 03 21:15:04.951797 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:04.951747 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-maas-api-tls\") pod \"51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91\" (UID: \"51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91\") " Nov 03 21:15:04.951866 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:04.951814 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-metrics-certs\") pod \"51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91\" (UID: \"51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91\") " Nov 03 21:15:04.951952 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:04.951931 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-openshift-service-ca\") pod \"51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91\" (UID: \"51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91\") " Nov 03 21:15:04.952389 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:04.952362 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-openshift-service-ca" pod "51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91" (UID: "51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:04.952609 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:04.952587 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:04.953904 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:04.953879 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-maas-api-tls" pod "51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91" (UID: "51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:04.954013 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:04.953937 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-kube-api-access-wr8q4" pod "51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91" (UID: "51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91"). InnerVolumeSpecName "kube-api-access-wr8q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:04.954013 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:04.953935 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-metrics-certs" pod "51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91" (UID: "51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:05.141497 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:05.140055 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:05.141497 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:05.140147 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:05.141497 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:05.140186 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr8q4\" (UniqueName: \"kubernetes.io/projected/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91-kube-api-access-wr8q4\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:05.854778 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:05.854746 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv" Nov 03 21:15:05.854778 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:05.854760 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv" event={"ID":"51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91","Type":"ContainerDied","Data":"d775f4ac7fe4b0871e57117e116015830382f59be486d51974f61eaf4bccdb8b"} Nov 03 21:15:05.855262 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:05.854829 5615 scope.go:122] "RemoveContainer" containerID="aa65af3220b554e1e7fef2e8e8f2a7980f29280f301546d4e5c7b937f211c644" Nov 03 21:15:06.011242 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:06.011204 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv"] Nov 03 21:15:06.058746 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:06.058686 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3337df-54bd8f479d-497kv"] Nov 03 21:15:07.653089 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:07.653049 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91" path="/var/lib/kubelet/pods/51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91/volumes" Nov 03 21:15:10.670231 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:15:10.670187 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:15:14.850646 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:14.850617 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-75947bbfb8-lnlqk" Nov 03 21:15:15.839221 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:15.839182 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg"] Nov 03 21:15:15.839871 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:15.839856 5615 cpu_manager.go:414] "containerMap: removing container" podUID="51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91" containerName="maas-api" Nov 03 21:15:15.839928 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:15.839874 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91" containerName="maas-api" Nov 03 21:15:15.840009 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:15.840000 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="51b02bb8-1b5a-4be2-92f5-8a4e3b50ef91" containerName="maas-api" Nov 03 21:15:15.843523 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:15.843492 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg" Nov 03 21:15:15.942177 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:15.942143 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-32f6e43c-serving-cert\"" Nov 03 21:15:15.947118 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:15.947069 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg"] Nov 03 21:15:15.977028 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:15.976987 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c156401-fe0c-4439-b1a8-7c9bdd23a624-metrics-certs\") pod \"maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg\" (UID: \"6c156401-fe0c-4439-b1a8-7c9bdd23a624\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg" Nov 03 21:15:15.977320 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:15.977287 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c156401-fe0c-4439-b1a8-7c9bdd23a624-maas-api-tls\") pod \"maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg\" (UID: \"6c156401-fe0c-4439-b1a8-7c9bdd23a624\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg" Nov 03 21:15:15.977391 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:15.977368 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c156401-fe0c-4439-b1a8-7c9bdd23a624-openshift-service-ca\") pod \"maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg\" (UID: \"6c156401-fe0c-4439-b1a8-7c9bdd23a624\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg" Nov 03 21:15:15.977453 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:15.977425 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kphjf\" (UniqueName: \"kubernetes.io/projected/6c156401-fe0c-4439-b1a8-7c9bdd23a624-kube-api-access-kphjf\") pod \"maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg\" (UID: \"6c156401-fe0c-4439-b1a8-7c9bdd23a624\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg" Nov 03 21:15:16.079287 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:16.079237 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c156401-fe0c-4439-b1a8-7c9bdd23a624-metrics-certs\") pod \"maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg\" (UID: \"6c156401-fe0c-4439-b1a8-7c9bdd23a624\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg" Nov 03 21:15:16.079698 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:16.079446 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c156401-fe0c-4439-b1a8-7c9bdd23a624-maas-api-tls\") pod \"maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg\" (UID: \"6c156401-fe0c-4439-b1a8-7c9bdd23a624\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg" Nov 03 21:15:16.079698 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:16.079508 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c156401-fe0c-4439-b1a8-7c9bdd23a624-openshift-service-ca\") pod \"maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg\" (UID: \"6c156401-fe0c-4439-b1a8-7c9bdd23a624\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg" Nov 03 21:15:16.079698 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:16.079563 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kphjf\" (UniqueName: \"kubernetes.io/projected/6c156401-fe0c-4439-b1a8-7c9bdd23a624-kube-api-access-kphjf\") pod \"maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg\" (UID: \"6c156401-fe0c-4439-b1a8-7c9bdd23a624\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg" Nov 03 21:15:16.080735 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:15:16.080700 5615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-serving-cert: secret "maas-api-e2e-mt-32f6e43c-serving-cert" not found Nov 03 21:15:16.081035 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:15:16.081017 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c156401-fe0c-4439-b1a8-7c9bdd23a624-maas-api-tls podName:6c156401-fe0c-4439-b1a8-7c9bdd23a624 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:16.580956614 +0000 UTC m=+1611.048735315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6c156401-fe0c-4439-b1a8-7c9bdd23a624-maas-api-tls") pod "maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg" (UID: "6c156401-fe0c-4439-b1a8-7c9bdd23a624") : secret "maas-api-e2e-mt-32f6e43c-serving-cert" not found Nov 03 21:15:16.107153 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:16.099282 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c156401-fe0c-4439-b1a8-7c9bdd23a624-openshift-service-ca\") pod \"maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg\" (UID: \"6c156401-fe0c-4439-b1a8-7c9bdd23a624\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg" Nov 03 21:15:16.152125 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:16.147403 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c156401-fe0c-4439-b1a8-7c9bdd23a624-metrics-certs\") pod \"maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg\" (UID: \"6c156401-fe0c-4439-b1a8-7c9bdd23a624\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg" Nov 03 21:15:16.168148 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:16.165799 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kphjf\" (UniqueName: \"kubernetes.io/projected/6c156401-fe0c-4439-b1a8-7c9bdd23a624-kube-api-access-kphjf\") pod \"maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg\" (UID: \"6c156401-fe0c-4439-b1a8-7c9bdd23a624\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg" Nov 03 21:15:16.600873 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:16.600831 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c156401-fe0c-4439-b1a8-7c9bdd23a624-maas-api-tls\") pod \"maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg\" (UID: \"6c156401-fe0c-4439-b1a8-7c9bdd23a624\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg" Nov 03 21:15:16.603328 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:16.603302 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c156401-fe0c-4439-b1a8-7c9bdd23a624-maas-api-tls\") pod \"maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg\" (UID: \"6c156401-fe0c-4439-b1a8-7c9bdd23a624\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg" Nov 03 21:15:16.771099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:16.767349 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg" Nov 03 21:15:17.330035 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:15:17.329999 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c156401_fe0c_4439_b1a8_7c9bdd23a624.slice/crio-2c70d9b09240279dcf601802d92c500a482905a2e5aaec8d444cccfebbb11dcc WatchSource:0}: Error finding container 2c70d9b09240279dcf601802d92c500a482905a2e5aaec8d444cccfebbb11dcc: Status 404 returned error can't find the container with id 2c70d9b09240279dcf601802d92c500a482905a2e5aaec8d444cccfebbb11dcc Nov 03 21:15:17.331402 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:17.331370 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg"] Nov 03 21:15:17.331649 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:17.331632 5615 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:15:17.963943 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:17.963887 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg" event={"ID":"6c156401-fe0c-4439-b1a8-7c9bdd23a624","Type":"ContainerStarted","Data":"2c70d9b09240279dcf601802d92c500a482905a2e5aaec8d444cccfebbb11dcc"} Nov 03 21:15:19.018565 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:19.018519 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg" event={"ID":"6c156401-fe0c-4439-b1a8-7c9bdd23a624","Type":"ContainerStarted","Data":"f2cc3d4302d9e7cfa7609348651f78e81005812a5d1c3f34dc1f9c744ec2ee24"} Nov 03 21:15:19.021905 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:19.021880 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg" Nov 03 21:15:19.132178 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:19.132125 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg" podStartSLOduration=3.147986319 podStartE2EDuration="4.132104222s" totalImagesPullingTime="984.117903ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:19.131455333 +0000 UTC m=+1613.599234056" watchObservedRunningTime="2026-11-03 21:15:19.132104222 +0000 UTC m=+1613.599882945" Nov 03 21:15:21.278588 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:21.278207 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq"] Nov 03 21:15:21.278588 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:21.278480 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" podUID="7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8" containerName="istio-proxy" containerID="cri-o://d8a08f375ad6ea2cb8202a455a89bd1628d641df2cd3a0b43f724b191349b71b" gracePeriod=30 Nov 03 21:15:26.039976 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.039941 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg" Nov 03 21:15:26.059243 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.059215 5615 scope.go:122] "RemoveContainer" containerID="28d10fc2e4fde65a1406a814a096bc131cfa2fd91d26109f5756b86de88e8602" Nov 03 21:15:26.244609 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.244573 5615 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:15:26.244775 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.244632 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" podUID="7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:15:26.719173 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.719140 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:15:26.823594 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.823559 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istio-envoy\") pod \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " Nov 03 21:15:26.823594 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.823599 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-workload-socket\") pod \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " Nov 03 21:15:26.823826 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.823624 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-kube-api-access-vc5w8\" (UniqueName: \"kubernetes.io/projected/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-kube-api-access-vc5w8\") pod \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " Nov 03 21:15:26.823826 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.823663 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istiod-ca-cert\") pod \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " Nov 03 21:15:26.823826 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.823688 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istio-token\" (UniqueName: \"kubernetes.io/projected/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istio-token\") pod \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " Nov 03 21:15:26.823826 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.823707 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-credential-socket\") pod \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " Nov 03 21:15:26.823826 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.823741 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istio-podinfo\") pod \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " Nov 03 21:15:26.823826 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.823806 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-workload-certs\") pod \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " Nov 03 21:15:26.824178 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.823930 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-workload-socket" pod "7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8" (UID: "7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:26.824178 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.824008 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istio-data\") pod \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\" (UID: \"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8\") " Nov 03 21:15:26.824178 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.824063 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-credential-socket" pod "7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8" (UID: "7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:26.824357 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.824318 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-workload-certs" pod "7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8" (UID: "7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:26.824415 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.824250 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istiod-ca-cert" pod "7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8" (UID: "7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:26.824467 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.824412 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istio-data" pod "7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8" (UID: "7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:26.824828 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.824807 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:26.824828 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.824827 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:26.824981 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.824836 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:26.824981 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.824845 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:26.824981 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.824853 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:26.826239 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.826212 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istio-podinfo" pod "7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8" (UID: "7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:15:26.826343 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.826299 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-kube-api-access-vc5w8" pod "7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8" (UID: "7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8"). InnerVolumeSpecName "kube-api-access-vc5w8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:26.826452 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.826432 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istio-token" pod "7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8" (UID: "7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:26.826510 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.826495 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istio-envoy" pod "7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8" (UID: "7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:26.925951 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.925917 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:26.925951 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.925952 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc5w8\" (UniqueName: \"kubernetes.io/projected/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-kube-api-access-vc5w8\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:26.926194 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.925967 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:26.926194 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:26.925976 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:27.079815 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:27.079773 5615 generic.go:356] "Generic (PLEG): container finished" podID="7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8" containerID="d8a08f375ad6ea2cb8202a455a89bd1628d641df2cd3a0b43f724b191349b71b" exitCode=0 Nov 03 21:15:27.079815 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:27.079818 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" Nov 03 21:15:27.080414 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:27.079861 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" event={"ID":"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8","Type":"ContainerDied","Data":"d8a08f375ad6ea2cb8202a455a89bd1628d641df2cd3a0b43f724b191349b71b"} Nov 03 21:15:27.080414 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:27.079920 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq" event={"ID":"7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8","Type":"ContainerDied","Data":"494efbd4a4575d34ad923a18c21e8a9fda42ce3bff1fc9127c8436ad7877ec11"} Nov 03 21:15:27.080414 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:27.079954 5615 scope.go:122] "RemoveContainer" containerID="d8a08f375ad6ea2cb8202a455a89bd1628d641df2cd3a0b43f724b191349b71b" Nov 03 21:15:27.096293 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:27.096263 5615 scope.go:122] "RemoveContainer" containerID="d8a08f375ad6ea2cb8202a455a89bd1628d641df2cd3a0b43f724b191349b71b" Nov 03 21:15:27.096912 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:15:27.096880 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8a08f375ad6ea2cb8202a455a89bd1628d641df2cd3a0b43f724b191349b71b\": container with ID starting with d8a08f375ad6ea2cb8202a455a89bd1628d641df2cd3a0b43f724b191349b71b not found: ID does not exist" containerID="d8a08f375ad6ea2cb8202a455a89bd1628d641df2cd3a0b43f724b191349b71b" Nov 03 21:15:27.097120 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:27.096928 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a08f375ad6ea2cb8202a455a89bd1628d641df2cd3a0b43f724b191349b71b"} err="failed to get container status \"d8a08f375ad6ea2cb8202a455a89bd1628d641df2cd3a0b43f724b191349b71b\": rpc error: code = NotFound desc = could not find container \"d8a08f375ad6ea2cb8202a455a89bd1628d641df2cd3a0b43f724b191349b71b\": container with ID starting with d8a08f375ad6ea2cb8202a455a89bd1628d641df2cd3a0b43f724b191349b71b not found: ID does not exist" Nov 03 21:15:27.103985 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:27.103958 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3e92e3-85f886c5dd-c5pzb"] Nov 03 21:15:27.237651 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:27.237613 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq"] Nov 03 21:15:27.243922 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:27.243888 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r5dkq"] Nov 03 21:15:27.732745 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:27.727494 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8" path="/var/lib/kubelet/pods/7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8/volumes" Nov 03 21:15:31.490841 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:31.490798 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-7468b6b967-4nh5h"] Nov 03 21:15:31.491689 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:31.491665 5615 cpu_manager.go:414] "containerMap: removing container" podUID="7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8" containerName="istio-proxy" Nov 03 21:15:31.491794 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:31.491690 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8" containerName="istio-proxy" Nov 03 21:15:31.491906 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:31.491892 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d1a6e0e-b6e7-4e5c-9ae2-738cdc0bd2c8" containerName="istio-proxy" Nov 03 21:15:31.496810 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:31.496788 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-7468b6b967-4nh5h" Nov 03 21:15:31.530597 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:31.530562 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzl8n\" (UniqueName: \"kubernetes.io/projected/79c93b60-376b-4176-86b9-e786ec8334b4-kube-api-access-hzl8n\") pod \"payload-pre-processing-e2e-ipp-a-66513c-7468b6b967-4nh5h\" (UID: \"79c93b60-376b-4176-86b9-e786ec8334b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-7468b6b967-4nh5h" Nov 03 21:15:31.530765 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:31.530610 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79c93b60-376b-4176-86b9-e786ec8334b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-66513c-7468b6b967-4nh5h\" (UID: \"79c93b60-376b-4176-86b9-e786ec8334b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-7468b6b967-4nh5h" Nov 03 21:15:31.548755 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:31.548718 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-7468b6b967-4nh5h"] Nov 03 21:15:31.598107 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:31.597346 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-66513c\"" Nov 03 21:15:31.598305 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:31.598168 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-66513c-dockercfg-tt2lc\"" Nov 03 21:15:31.643100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:31.639041 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzl8n\" (UniqueName: \"kubernetes.io/projected/79c93b60-376b-4176-86b9-e786ec8334b4-kube-api-access-hzl8n\") pod \"payload-pre-processing-e2e-ipp-a-66513c-7468b6b967-4nh5h\" (UID: \"79c93b60-376b-4176-86b9-e786ec8334b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-7468b6b967-4nh5h" Nov 03 21:15:31.643100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:31.639133 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79c93b60-376b-4176-86b9-e786ec8334b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-66513c-7468b6b967-4nh5h\" (UID: \"79c93b60-376b-4176-86b9-e786ec8334b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-7468b6b967-4nh5h" Nov 03 21:15:31.643100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:31.640325 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79c93b60-376b-4176-86b9-e786ec8334b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-66513c-7468b6b967-4nh5h\" (UID: \"79c93b60-376b-4176-86b9-e786ec8334b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-7468b6b967-4nh5h" Nov 03 21:15:31.725097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:31.723231 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzl8n\" (UniqueName: \"kubernetes.io/projected/79c93b60-376b-4176-86b9-e786ec8334b4-kube-api-access-hzl8n\") pod \"payload-pre-processing-e2e-ipp-a-66513c-7468b6b967-4nh5h\" (UID: \"79c93b60-376b-4176-86b9-e786ec8334b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-7468b6b967-4nh5h" Nov 03 21:15:31.812525 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:31.812421 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-7468b6b967-4nh5h" Nov 03 21:15:32.104735 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:32.104628 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-3e92e3-85f886c5dd-c5pzb" podUID="50f16bde-5aa7-46f6-b801-68d84fa02ae3" containerName="payload-processing" containerID="cri-o://669bbf139390438bd069da8c01f1c8437a8e0bcfc021d293f1b7f7cb07200765" gracePeriod=25 Nov 03 21:15:32.194614 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:32.194578 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-7468b6b967-4nh5h"] Nov 03 21:15:32.196475 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:15:32.196441 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c93b60_376b_4176_86b9_e786ec8334b4.slice/crio-e681158fef749733b4f55e4be1079a9738f1b57a528ae490a789c0095cc3a932 WatchSource:0}: Error finding container e681158fef749733b4f55e4be1079a9738f1b57a528ae490a789c0095cc3a932: Status 404 returned error can't find the container with id e681158fef749733b4f55e4be1079a9738f1b57a528ae490a789c0095cc3a932 Nov 03 21:15:32.378387 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:32.378351 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3e92e3-85f886c5dd-c5pzb" Nov 03 21:15:32.451724 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:32.451689 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50f16bde-5aa7-46f6-b801-68d84fa02ae3-kube-api-access-wns28\" (UniqueName: \"kubernetes.io/projected/50f16bde-5aa7-46f6-b801-68d84fa02ae3-kube-api-access-wns28\") pod \"50f16bde-5aa7-46f6-b801-68d84fa02ae3\" (UID: \"50f16bde-5aa7-46f6-b801-68d84fa02ae3\") " Nov 03 21:15:32.451906 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:32.451802 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50f16bde-5aa7-46f6-b801-68d84fa02ae3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f16bde-5aa7-46f6-b801-68d84fa02ae3-plugins-config-volume\") pod \"50f16bde-5aa7-46f6-b801-68d84fa02ae3\" (UID: \"50f16bde-5aa7-46f6-b801-68d84fa02ae3\") " Nov 03 21:15:32.452208 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:32.452184 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f16bde-5aa7-46f6-b801-68d84fa02ae3-plugins-config-volume" pod "50f16bde-5aa7-46f6-b801-68d84fa02ae3" (UID: "50f16bde-5aa7-46f6-b801-68d84fa02ae3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:32.452681 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:32.452659 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f16bde-5aa7-46f6-b801-68d84fa02ae3-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:32.454015 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:32.453995 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f16bde-5aa7-46f6-b801-68d84fa02ae3-kube-api-access-wns28" pod "50f16bde-5aa7-46f6-b801-68d84fa02ae3" (UID: "50f16bde-5aa7-46f6-b801-68d84fa02ae3"). InnerVolumeSpecName "kube-api-access-wns28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:32.555459 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:32.555419 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wns28\" (UniqueName: \"kubernetes.io/projected/50f16bde-5aa7-46f6-b801-68d84fa02ae3-kube-api-access-wns28\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:33.134427 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:33.134319 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-7468b6b967-4nh5h" event={"ID":"79c93b60-376b-4176-86b9-e786ec8334b4","Type":"ContainerStarted","Data":"ea8dee4e3d0da701a8b2aeb41f2a6b9448f8341559353f7732a807259a0646d4"} Nov 03 21:15:33.134595 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:33.134499 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-7468b6b967-4nh5h" Nov 03 21:15:33.134595 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:33.134530 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-7468b6b967-4nh5h" event={"ID":"79c93b60-376b-4176-86b9-e786ec8334b4","Type":"ContainerStarted","Data":"e681158fef749733b4f55e4be1079a9738f1b57a528ae490a789c0095cc3a932"} Nov 03 21:15:33.135828 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:33.135801 5615 generic.go:356] "Generic (PLEG): container finished" podID="50f16bde-5aa7-46f6-b801-68d84fa02ae3" containerID="669bbf139390438bd069da8c01f1c8437a8e0bcfc021d293f1b7f7cb07200765" exitCode=0 Nov 03 21:15:33.135944 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:33.135840 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3e92e3-85f886c5dd-c5pzb" event={"ID":"50f16bde-5aa7-46f6-b801-68d84fa02ae3","Type":"ContainerDied","Data":"669bbf139390438bd069da8c01f1c8437a8e0bcfc021d293f1b7f7cb07200765"} Nov 03 21:15:33.135944 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:33.135849 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3e92e3-85f886c5dd-c5pzb" Nov 03 21:15:33.135944 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:33.135877 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3e92e3-85f886c5dd-c5pzb" event={"ID":"50f16bde-5aa7-46f6-b801-68d84fa02ae3","Type":"ContainerDied","Data":"3a7b96065a6ae090bf7dbf2022961f7e2008f3a60798c86b326d5e93bcf6a39f"} Nov 03 21:15:33.135944 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:33.135901 5615 scope.go:122] "RemoveContainer" containerID="669bbf139390438bd069da8c01f1c8437a8e0bcfc021d293f1b7f7cb07200765" Nov 03 21:15:33.146141 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:33.146120 5615 scope.go:122] "RemoveContainer" containerID="669bbf139390438bd069da8c01f1c8437a8e0bcfc021d293f1b7f7cb07200765" Nov 03 21:15:33.146478 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:15:33.146449 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"669bbf139390438bd069da8c01f1c8437a8e0bcfc021d293f1b7f7cb07200765\": container with ID starting with 669bbf139390438bd069da8c01f1c8437a8e0bcfc021d293f1b7f7cb07200765 not found: ID does not exist" containerID="669bbf139390438bd069da8c01f1c8437a8e0bcfc021d293f1b7f7cb07200765" Nov 03 21:15:33.146552 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:33.146494 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"669bbf139390438bd069da8c01f1c8437a8e0bcfc021d293f1b7f7cb07200765"} err="failed to get container status \"669bbf139390438bd069da8c01f1c8437a8e0bcfc021d293f1b7f7cb07200765\": rpc error: code = NotFound desc = could not find container \"669bbf139390438bd069da8c01f1c8437a8e0bcfc021d293f1b7f7cb07200765\": container with ID starting with 669bbf139390438bd069da8c01f1c8437a8e0bcfc021d293f1b7f7cb07200765 not found: ID does not exist" Nov 03 21:15:33.246489 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:33.246434 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-7468b6b967-4nh5h" podStartSLOduration=2.246418988 podStartE2EDuration="2.246418988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:33.24460454 +0000 UTC m=+1627.712383262" watchObservedRunningTime="2026-11-03 21:15:33.246418988 +0000 UTC m=+1627.714197714" Nov 03 21:15:33.350629 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:33.350590 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3e92e3-85f886c5dd-c5pzb"] Nov 03 21:15:33.357119 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:33.357086 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3e92e3-85f886c5dd-c5pzb"] Nov 03 21:15:33.653791 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:33.653755 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50f16bde-5aa7-46f6-b801-68d84fa02ae3" path="/var/lib/kubelet/pods/50f16bde-5aa7-46f6-b801-68d84fa02ae3/volumes" Nov 03 21:15:36.238212 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:36.237719 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6"] Nov 03 21:15:36.238212 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:36.238030 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" podUID="eefda059-ea79-40bb-9304-12ab35450ab6" containerName="istio-proxy" containerID="cri-o://084ff384b0a93912052fba6f528f15de3aa18b750e899028f80d1779249e0028" gracePeriod=30 Nov 03 21:15:37.014331 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:37.014287 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0f8726-6878775966-spjk6"] Nov 03 21:15:37.146102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:37.143339 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-69f498847-h225h"] Nov 03 21:15:38.321816 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.321783 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc"] Nov 03 21:15:38.322505 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.322488 5615 cpu_manager.go:414] "containerMap: removing container" podUID="50f16bde-5aa7-46f6-b801-68d84fa02ae3" containerName="payload-processing" Nov 03 21:15:38.322570 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.322505 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50f16bde-5aa7-46f6-b801-68d84fa02ae3" containerName="payload-processing" Nov 03 21:15:38.322615 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.322594 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="50f16bde-5aa7-46f6-b801-68d84fa02ae3" containerName="payload-processing" Nov 03 21:15:38.327724 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.327700 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:38.377697 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.377662 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-9232e5-openshift-default-dockercfg-sxpwt\"" Nov 03 21:15:38.405644 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.405606 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-istio-envoy\") pod \"e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:38.405818 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.405654 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1310ac80-c9fa-4498-8bad-6be0ded4c495-istiod-ca-cert\") pod \"e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:38.405818 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.405741 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1310ac80-c9fa-4498-8bad-6be0ded4c495-istio-podinfo\") pod \"e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:38.405818 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.405772 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1310ac80-c9fa-4498-8bad-6be0ded4c495-istio-token\") pod \"e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:38.405925 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.405853 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-istio-data\") pod \"e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:38.405974 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.405958 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-workload-certs\") pod \"e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:38.406021 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.406009 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s82bm\" (UniqueName: \"kubernetes.io/projected/1310ac80-c9fa-4498-8bad-6be0ded4c495-kube-api-access-s82bm\") pod \"e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:38.406055 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.406034 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-credential-socket\") pod \"e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:38.406195 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.406182 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-workload-socket\") pod \"e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:38.423429 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.423395 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc"] Nov 03 21:15:38.507688 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.507648 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s82bm\" (UniqueName: \"kubernetes.io/projected/1310ac80-c9fa-4498-8bad-6be0ded4c495-kube-api-access-s82bm\") pod \"e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:38.507688 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.507691 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-credential-socket\") pod \"e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:38.507956 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.507776 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-workload-socket\") pod \"e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:38.507956 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.507846 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-istio-envoy\") pod \"e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:38.507956 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.507868 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1310ac80-c9fa-4498-8bad-6be0ded4c495-istiod-ca-cert\") pod \"e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:38.508166 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.508006 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1310ac80-c9fa-4498-8bad-6be0ded4c495-istio-podinfo\") pod \"e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:38.508166 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.508061 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1310ac80-c9fa-4498-8bad-6be0ded4c495-istio-token\") pod \"e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:38.508166 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.508134 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-istio-data\") pod \"e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:38.508319 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.508230 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-credential-socket\") pod \"e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:38.508319 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.508252 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-workload-certs\") pod \"e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:38.508319 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.508284 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-workload-socket\") pod \"e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:38.508636 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.508613 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-workload-certs\") pod \"e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:38.508714 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.508658 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1310ac80-c9fa-4498-8bad-6be0ded4c495-istiod-ca-cert\") pod \"e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:38.508714 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.508655 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-istio-data\") pod \"e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:38.510295 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.510272 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-istio-envoy\") pod \"e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:38.510782 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.510761 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1310ac80-c9fa-4498-8bad-6be0ded4c495-istio-podinfo\") pod \"e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:38.618367 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.609245 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s82bm\" (UniqueName: \"kubernetes.io/projected/1310ac80-c9fa-4498-8bad-6be0ded4c495-kube-api-access-s82bm\") pod \"e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:38.625712 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.625634 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1310ac80-c9fa-4498-8bad-6be0ded4c495-istio-token\") pod \"e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:38.653192 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:38.646899 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:39.247022 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:39.246993 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc"] Nov 03 21:15:39.249351 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:15:39.249321 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1310ac80_c9fa_4498_8bad_6be0ded4c495.slice/crio-ef256ae459c49c71051a58723abc052ac912ca205589605aa7daa817ca1da15c WatchSource:0}: Error finding container ef256ae459c49c71051a58723abc052ac912ca205589605aa7daa817ca1da15c: Status 404 returned error can't find the container with id ef256ae459c49c71051a58723abc052ac912ca205589605aa7daa817ca1da15c Nov 03 21:15:39.251936 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:39.251896 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:15:39.252024 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:39.251984 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:15:39.252108 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:39.252028 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:15:39.322553 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:39.322518 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" event={"ID":"1310ac80-c9fa-4498-8bad-6be0ded4c495","Type":"ContainerStarted","Data":"dc18fcaae9950171f61627ff17264f9ac5dd80f3e63812ec6c525f146a3b9a08"} Nov 03 21:15:39.322898 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:39.322582 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" event={"ID":"1310ac80-c9fa-4498-8bad-6be0ded4c495","Type":"ContainerStarted","Data":"ef256ae459c49c71051a58723abc052ac912ca205589605aa7daa817ca1da15c"} Nov 03 21:15:39.440257 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:39.440203 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" podStartSLOduration=1.440189423 podStartE2EDuration="1.440189423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:39.437275412 +0000 UTC m=+1633.905054147" watchObservedRunningTime="2026-11-03 21:15:39.440189423 +0000 UTC m=+1633.907968145" Nov 03 21:15:39.654528 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:39.654450 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:40.620668 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:15:40.620629 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:15:40.659378 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:40.659344 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:41.348230 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:41.348190 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:41.349334 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:41.349308 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:15:41.664137 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:41.664110 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:15:41.803437 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:41.803396 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-workload-socket\") pod \"eefda059-ea79-40bb-9304-12ab35450ab6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " Nov 03 21:15:41.803618 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:41.803519 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-istio-data\") pod \"eefda059-ea79-40bb-9304-12ab35450ab6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " Nov 03 21:15:41.803618 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:41.803573 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eefda059-ea79-40bb-9304-12ab35450ab6-kube-api-access-qrr4t\" (UniqueName: \"kubernetes.io/projected/eefda059-ea79-40bb-9304-12ab35450ab6-kube-api-access-qrr4t\") pod \"eefda059-ea79-40bb-9304-12ab35450ab6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " Nov 03 21:15:41.803734 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:41.803615 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eefda059-ea79-40bb-9304-12ab35450ab6-istio-token\" (UniqueName: \"kubernetes.io/projected/eefda059-ea79-40bb-9304-12ab35450ab6-istio-token\") pod \"eefda059-ea79-40bb-9304-12ab35450ab6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " Nov 03 21:15:41.803734 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:41.803674 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eefda059-ea79-40bb-9304-12ab35450ab6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eefda059-ea79-40bb-9304-12ab35450ab6-istio-podinfo\") pod \"eefda059-ea79-40bb-9304-12ab35450ab6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " Nov 03 21:15:41.803734 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:41.803729 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-workload-certs\") pod \"eefda059-ea79-40bb-9304-12ab35450ab6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " Nov 03 21:15:41.803887 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:41.803770 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eefda059-ea79-40bb-9304-12ab35450ab6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eefda059-ea79-40bb-9304-12ab35450ab6-istiod-ca-cert\") pod \"eefda059-ea79-40bb-9304-12ab35450ab6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " Nov 03 21:15:41.803887 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:41.803752 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-workload-socket" pod "eefda059-ea79-40bb-9304-12ab35450ab6" (UID: "eefda059-ea79-40bb-9304-12ab35450ab6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:41.803887 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:41.803836 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-istio-data" pod "eefda059-ea79-40bb-9304-12ab35450ab6" (UID: "eefda059-ea79-40bb-9304-12ab35450ab6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:41.803887 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:41.803846 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-istio-envoy\") pod \"eefda059-ea79-40bb-9304-12ab35450ab6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " Nov 03 21:15:41.804104 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:41.803967 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-credential-socket\") pod \"eefda059-ea79-40bb-9304-12ab35450ab6\" (UID: \"eefda059-ea79-40bb-9304-12ab35450ab6\") " Nov 03 21:15:41.805641 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:41.805234 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:41.805641 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:41.805458 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eefda059-ea79-40bb-9304-12ab35450ab6-istiod-ca-cert" pod "eefda059-ea79-40bb-9304-12ab35450ab6" (UID: "eefda059-ea79-40bb-9304-12ab35450ab6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:41.805641 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:41.805471 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:41.805641 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:41.805515 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-credential-socket" pod "eefda059-ea79-40bb-9304-12ab35450ab6" (UID: "eefda059-ea79-40bb-9304-12ab35450ab6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:41.805641 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:41.805586 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-workload-certs" pod "eefda059-ea79-40bb-9304-12ab35450ab6" (UID: "eefda059-ea79-40bb-9304-12ab35450ab6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:41.807018 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:41.806985 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eefda059-ea79-40bb-9304-12ab35450ab6-kube-api-access-qrr4t" pod "eefda059-ea79-40bb-9304-12ab35450ab6" (UID: "eefda059-ea79-40bb-9304-12ab35450ab6"). InnerVolumeSpecName "kube-api-access-qrr4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:41.808536 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:41.808509 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-istio-envoy" pod "eefda059-ea79-40bb-9304-12ab35450ab6" (UID: "eefda059-ea79-40bb-9304-12ab35450ab6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:41.808693 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:41.808671 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eefda059-ea79-40bb-9304-12ab35450ab6-istio-podinfo" pod "eefda059-ea79-40bb-9304-12ab35450ab6" (UID: "eefda059-ea79-40bb-9304-12ab35450ab6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:15:41.809041 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:41.809017 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eefda059-ea79-40bb-9304-12ab35450ab6-istio-token" pod "eefda059-ea79-40bb-9304-12ab35450ab6" (UID: "eefda059-ea79-40bb-9304-12ab35450ab6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:41.906818 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:41.906721 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:41.906818 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:41.906758 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eefda059-ea79-40bb-9304-12ab35450ab6-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:41.906818 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:41.906771 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:41.906818 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:41.906783 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefda059-ea79-40bb-9304-12ab35450ab6-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:41.906818 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:41.906796 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrr4t\" (UniqueName: \"kubernetes.io/projected/eefda059-ea79-40bb-9304-12ab35450ab6-kube-api-access-qrr4t\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:41.906818 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:41.906809 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eefda059-ea79-40bb-9304-12ab35450ab6-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:41.906818 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:41.906821 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eefda059-ea79-40bb-9304-12ab35450ab6-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:42.015956 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.015909 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-6878775966-spjk6" podUID="e0b123ee-6eab-443c-9a71-af860f6f5ace" containerName="payload-processing" containerID="cri-o://39857459fc21a9a05564850762a487267870970f0472af7c476f02c8a2c5405f" gracePeriod=25 Nov 03 21:15:42.144444 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.144395 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-69f498847-h225h" podUID="ce818536-2d2c-4aa1-865d-1921957f995e" containerName="payload-pre-processing" containerID="cri-o://b53b6f461cf984259af3fbae8b9d489038c41a52f930df4d4478582022aa2cfd" gracePeriod=25 Nov 03 21:15:42.291322 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.291295 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-6878775966-spjk6" Nov 03 21:15:42.363542 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.363503 5615 generic.go:356] "Generic (PLEG): container finished" podID="eefda059-ea79-40bb-9304-12ab35450ab6" containerID="084ff384b0a93912052fba6f528f15de3aa18b750e899028f80d1779249e0028" exitCode=0 Nov 03 21:15:42.363718 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.363563 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" Nov 03 21:15:42.363718 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.363589 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" event={"ID":"eefda059-ea79-40bb-9304-12ab35450ab6","Type":"ContainerDied","Data":"084ff384b0a93912052fba6f528f15de3aa18b750e899028f80d1779249e0028"} Nov 03 21:15:42.363978 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.363953 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6" event={"ID":"eefda059-ea79-40bb-9304-12ab35450ab6","Type":"ContainerDied","Data":"651a866c50152f811689829e9e4f866457bc43ee82294fdaa025eb4d910f7af8"} Nov 03 21:15:42.364068 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.363995 5615 scope.go:122] "RemoveContainer" containerID="084ff384b0a93912052fba6f528f15de3aa18b750e899028f80d1779249e0028" Nov 03 21:15:42.365910 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.365887 5615 generic.go:356] "Generic (PLEG): container finished" podID="e0b123ee-6eab-443c-9a71-af860f6f5ace" containerID="39857459fc21a9a05564850762a487267870970f0472af7c476f02c8a2c5405f" exitCode=0 Nov 03 21:15:42.366004 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.365921 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-6878775966-spjk6" event={"ID":"e0b123ee-6eab-443c-9a71-af860f6f5ace","Type":"ContainerDied","Data":"39857459fc21a9a05564850762a487267870970f0472af7c476f02c8a2c5405f"} Nov 03 21:15:42.366004 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.365956 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-6878775966-spjk6" Nov 03 21:15:42.366004 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.365974 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-6878775966-spjk6" event={"ID":"e0b123ee-6eab-443c-9a71-af860f6f5ace","Type":"ContainerDied","Data":"6fc3f7bc47d626041e7fa4558537ba39c515fefe96cacca3327591ac281ca6e1"} Nov 03 21:15:42.368183 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.368160 5615 generic.go:356] "Generic (PLEG): container finished" podID="ce818536-2d2c-4aa1-865d-1921957f995e" containerID="b53b6f461cf984259af3fbae8b9d489038c41a52f930df4d4478582022aa2cfd" exitCode=0 Nov 03 21:15:42.368264 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.368222 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-69f498847-h225h" event={"ID":"ce818536-2d2c-4aa1-865d-1921957f995e","Type":"ContainerDied","Data":"b53b6f461cf984259af3fbae8b9d489038c41a52f930df4d4478582022aa2cfd"} Nov 03 21:15:42.380137 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.380067 5615 scope.go:122] "RemoveContainer" containerID="084ff384b0a93912052fba6f528f15de3aa18b750e899028f80d1779249e0028" Nov 03 21:15:42.380656 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:15:42.380508 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"084ff384b0a93912052fba6f528f15de3aa18b750e899028f80d1779249e0028\": container with ID starting with 084ff384b0a93912052fba6f528f15de3aa18b750e899028f80d1779249e0028 not found: ID does not exist" containerID="084ff384b0a93912052fba6f528f15de3aa18b750e899028f80d1779249e0028" Nov 03 21:15:42.380656 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.380547 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"084ff384b0a93912052fba6f528f15de3aa18b750e899028f80d1779249e0028"} err="failed to get container status \"084ff384b0a93912052fba6f528f15de3aa18b750e899028f80d1779249e0028\": rpc error: code = NotFound desc = could not find container \"084ff384b0a93912052fba6f528f15de3aa18b750e899028f80d1779249e0028\": container with ID starting with 084ff384b0a93912052fba6f528f15de3aa18b750e899028f80d1779249e0028 not found: ID does not exist" Nov 03 21:15:42.380656 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.380572 5615 scope.go:122] "RemoveContainer" containerID="39857459fc21a9a05564850762a487267870970f0472af7c476f02c8a2c5405f" Nov 03 21:15:42.401654 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.395037 5615 scope.go:122] "RemoveContainer" containerID="39857459fc21a9a05564850762a487267870970f0472af7c476f02c8a2c5405f" Nov 03 21:15:42.401654 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:15:42.395416 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39857459fc21a9a05564850762a487267870970f0472af7c476f02c8a2c5405f\": container with ID starting with 39857459fc21a9a05564850762a487267870970f0472af7c476f02c8a2c5405f not found: ID does not exist" containerID="39857459fc21a9a05564850762a487267870970f0472af7c476f02c8a2c5405f" Nov 03 21:15:42.401654 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.395456 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39857459fc21a9a05564850762a487267870970f0472af7c476f02c8a2c5405f"} err="failed to get container status \"39857459fc21a9a05564850762a487267870970f0472af7c476f02c8a2c5405f\": rpc error: code = NotFound desc = could not find container \"39857459fc21a9a05564850762a487267870970f0472af7c476f02c8a2c5405f\": container with ID starting with 39857459fc21a9a05564850762a487267870970f0472af7c476f02c8a2c5405f not found: ID does not exist" Nov 03 21:15:42.417305 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.417266 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0b123ee-6eab-443c-9a71-af860f6f5ace-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0b123ee-6eab-443c-9a71-af860f6f5ace-plugins-config-volume\") pod \"e0b123ee-6eab-443c-9a71-af860f6f5ace\" (UID: \"e0b123ee-6eab-443c-9a71-af860f6f5ace\") " Nov 03 21:15:42.417486 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.417466 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0b123ee-6eab-443c-9a71-af860f6f5ace-kube-api-access-trm8x\" (UniqueName: \"kubernetes.io/projected/e0b123ee-6eab-443c-9a71-af860f6f5ace-kube-api-access-trm8x\") pod \"e0b123ee-6eab-443c-9a71-af860f6f5ace\" (UID: \"e0b123ee-6eab-443c-9a71-af860f6f5ace\") " Nov 03 21:15:42.417645 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.417622 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0b123ee-6eab-443c-9a71-af860f6f5ace-plugins-config-volume" pod "e0b123ee-6eab-443c-9a71-af860f6f5ace" (UID: "e0b123ee-6eab-443c-9a71-af860f6f5ace"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:42.418122 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.418062 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0b123ee-6eab-443c-9a71-af860f6f5ace-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:42.420616 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.420554 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b123ee-6eab-443c-9a71-af860f6f5ace-kube-api-access-trm8x" pod "e0b123ee-6eab-443c-9a71-af860f6f5ace" (UID: "e0b123ee-6eab-443c-9a71-af860f6f5ace"). InnerVolumeSpecName "kube-api-access-trm8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:42.468199 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.468170 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-69f498847-h225h" Nov 03 21:15:42.519767 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.519734 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trm8x\" (UniqueName: \"kubernetes.io/projected/e0b123ee-6eab-443c-9a71-af860f6f5ace-kube-api-access-trm8x\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:42.566400 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.566363 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6"] Nov 03 21:15:42.573150 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.573120 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-r4kt6"] Nov 03 21:15:42.620650 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.620595 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce818536-2d2c-4aa1-865d-1921957f995e-kube-api-access-w8bpf\" (UniqueName: \"kubernetes.io/projected/ce818536-2d2c-4aa1-865d-1921957f995e-kube-api-access-w8bpf\") pod \"ce818536-2d2c-4aa1-865d-1921957f995e\" (UID: \"ce818536-2d2c-4aa1-865d-1921957f995e\") " Nov 03 21:15:42.620842 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.620820 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce818536-2d2c-4aa1-865d-1921957f995e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce818536-2d2c-4aa1-865d-1921957f995e-plugins-config-volume\") pod \"ce818536-2d2c-4aa1-865d-1921957f995e\" (UID: \"ce818536-2d2c-4aa1-865d-1921957f995e\") " Nov 03 21:15:42.621216 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.621190 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce818536-2d2c-4aa1-865d-1921957f995e-plugins-config-volume" pod "ce818536-2d2c-4aa1-865d-1921957f995e" (UID: "ce818536-2d2c-4aa1-865d-1921957f995e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:42.621459 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.621438 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce818536-2d2c-4aa1-865d-1921957f995e-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:42.622769 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.622746 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce818536-2d2c-4aa1-865d-1921957f995e-kube-api-access-w8bpf" pod "ce818536-2d2c-4aa1-865d-1921957f995e" (UID: "ce818536-2d2c-4aa1-865d-1921957f995e"). InnerVolumeSpecName "kube-api-access-w8bpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:42.722831 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.722791 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8bpf\" (UniqueName: \"kubernetes.io/projected/ce818536-2d2c-4aa1-865d-1921957f995e-kube-api-access-w8bpf\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:42.794728 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.794668 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0f8726-6878775966-spjk6"] Nov 03 21:15:42.796759 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:42.796731 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0f8726-6878775966-spjk6"] Nov 03 21:15:43.377514 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:43.377482 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-69f498847-h225h" Nov 03 21:15:43.377675 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:43.377518 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-69f498847-h225h" event={"ID":"ce818536-2d2c-4aa1-865d-1921957f995e","Type":"ContainerDied","Data":"514f15732b2c5d2c9d4384878849cba3bb2361bedb04607ece36bec7e68ecacc"} Nov 03 21:15:43.377675 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:43.377583 5615 scope.go:122] "RemoveContainer" containerID="b53b6f461cf984259af3fbae8b9d489038c41a52f930df4d4478582022aa2cfd" Nov 03 21:15:43.535911 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:43.535878 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-69f498847-h225h"] Nov 03 21:15:43.583107 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:43.583059 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-69f498847-h225h"] Nov 03 21:15:43.653400 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:43.653314 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce818536-2d2c-4aa1-865d-1921957f995e" path="/var/lib/kubelet/pods/ce818536-2d2c-4aa1-865d-1921957f995e/volumes" Nov 03 21:15:43.653767 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:43.653753 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0b123ee-6eab-443c-9a71-af860f6f5ace" path="/var/lib/kubelet/pods/e0b123ee-6eab-443c-9a71-af860f6f5ace/volumes" Nov 03 21:15:43.654224 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:43.654211 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eefda059-ea79-40bb-9304-12ab35450ab6" path="/var/lib/kubelet/pods/eefda059-ea79-40bb-9304-12ab35450ab6/volumes" Nov 03 21:15:44.146363 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:44.146331 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-7468b6b967-4nh5h" Nov 03 21:15:46.082161 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.082127 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7"] Nov 03 21:15:46.083763 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.083728 5615 cpu_manager.go:414] "containerMap: removing container" podUID="ce818536-2d2c-4aa1-865d-1921957f995e" containerName="payload-pre-processing" Nov 03 21:15:46.091248 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.091210 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce818536-2d2c-4aa1-865d-1921957f995e" containerName="payload-pre-processing" Nov 03 21:15:46.091567 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.091542 5615 cpu_manager.go:414] "containerMap: removing container" podUID="eefda059-ea79-40bb-9304-12ab35450ab6" containerName="istio-proxy" Nov 03 21:15:46.091783 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.091763 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eefda059-ea79-40bb-9304-12ab35450ab6" containerName="istio-proxy" Nov 03 21:15:46.091930 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.091915 5615 cpu_manager.go:414] "containerMap: removing container" podUID="e0b123ee-6eab-443c-9a71-af860f6f5ace" containerName="payload-processing" Nov 03 21:15:46.092026 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.092010 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0b123ee-6eab-443c-9a71-af860f6f5ace" containerName="payload-processing" Nov 03 21:15:46.092355 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.092339 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0b123ee-6eab-443c-9a71-af860f6f5ace" containerName="payload-processing" Nov 03 21:15:46.092476 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.092463 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce818536-2d2c-4aa1-865d-1921957f995e" containerName="payload-pre-processing" Nov 03 21:15:46.092576 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.092565 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="eefda059-ea79-40bb-9304-12ab35450ab6" containerName="istio-proxy" Nov 03 21:15:46.101474 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.101444 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7" Nov 03 21:15:46.143094 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.143028 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7"] Nov 03 21:15:46.186465 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.185980 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-812243-serving-cert\"" Nov 03 21:15:46.226704 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.223052 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nclpv\" (UniqueName: \"kubernetes.io/projected/c1d863b8-3685-4948-b14b-9da2377616ff-kube-api-access-nclpv\") pod \"maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7\" (UID: \"c1d863b8-3685-4948-b14b-9da2377616ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7" Nov 03 21:15:46.226704 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.223195 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1d863b8-3685-4948-b14b-9da2377616ff-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7\" (UID: \"c1d863b8-3685-4948-b14b-9da2377616ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7" Nov 03 21:15:46.226704 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.223254 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1d863b8-3685-4948-b14b-9da2377616ff-metrics-certs\") pod \"maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7\" (UID: \"c1d863b8-3685-4948-b14b-9da2377616ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7" Nov 03 21:15:46.226704 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.223293 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1d863b8-3685-4948-b14b-9da2377616ff-maas-api-tls\") pod \"maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7\" (UID: \"c1d863b8-3685-4948-b14b-9da2377616ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7" Nov 03 21:15:46.324418 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.324381 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nclpv\" (UniqueName: \"kubernetes.io/projected/c1d863b8-3685-4948-b14b-9da2377616ff-kube-api-access-nclpv\") pod \"maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7\" (UID: \"c1d863b8-3685-4948-b14b-9da2377616ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7" Nov 03 21:15:46.324579 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.324492 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1d863b8-3685-4948-b14b-9da2377616ff-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7\" (UID: \"c1d863b8-3685-4948-b14b-9da2377616ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7" Nov 03 21:15:46.324579 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.324558 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1d863b8-3685-4948-b14b-9da2377616ff-metrics-certs\") pod \"maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7\" (UID: \"c1d863b8-3685-4948-b14b-9da2377616ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7" Nov 03 21:15:46.324711 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.324602 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1d863b8-3685-4948-b14b-9da2377616ff-maas-api-tls\") pod \"maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7\" (UID: \"c1d863b8-3685-4948-b14b-9da2377616ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7" Nov 03 21:15:46.325855 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.325827 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1d863b8-3685-4948-b14b-9da2377616ff-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7\" (UID: \"c1d863b8-3685-4948-b14b-9da2377616ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7" Nov 03 21:15:46.327562 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.327375 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1d863b8-3685-4948-b14b-9da2377616ff-maas-api-tls\") pod \"maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7\" (UID: \"c1d863b8-3685-4948-b14b-9da2377616ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7" Nov 03 21:15:46.329511 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.328050 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1d863b8-3685-4948-b14b-9da2377616ff-metrics-certs\") pod \"maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7\" (UID: \"c1d863b8-3685-4948-b14b-9da2377616ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7" Nov 03 21:15:46.345132 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.345002 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9232e5-7b98c54cc4-ngvtb"] Nov 03 21:15:46.350577 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.350545 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9232e5-7b98c54cc4-ngvtb" Nov 03 21:15:46.385017 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.384983 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nclpv\" (UniqueName: \"kubernetes.io/projected/c1d863b8-3685-4948-b14b-9da2377616ff-kube-api-access-nclpv\") pod \"maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7\" (UID: \"c1d863b8-3685-4948-b14b-9da2377616ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7" Nov 03 21:15:46.400581 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.400549 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9232e5\"" Nov 03 21:15:46.400581 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.400588 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9232e5-dockercfg-q62r2\"" Nov 03 21:15:46.401921 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.401895 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9232e5-7b98c54cc4-ngvtb"] Nov 03 21:15:46.425515 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.425475 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7906cbf-9a94-4fe8-952d-48fed7a1cd3e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9232e5-7b98c54cc4-ngvtb\" (UID: \"a7906cbf-9a94-4fe8-952d-48fed7a1cd3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9232e5-7b98c54cc4-ngvtb" Nov 03 21:15:46.425711 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.425558 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm7rn\" (UniqueName: \"kubernetes.io/projected/a7906cbf-9a94-4fe8-952d-48fed7a1cd3e-kube-api-access-mm7rn\") pod \"payload-processing-e2e-ipp-b-9232e5-7b98c54cc4-ngvtb\" (UID: \"a7906cbf-9a94-4fe8-952d-48fed7a1cd3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9232e5-7b98c54cc4-ngvtb" Nov 03 21:15:46.485619 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.485582 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7" Nov 03 21:15:46.545123 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.537503 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7906cbf-9a94-4fe8-952d-48fed7a1cd3e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9232e5-7b98c54cc4-ngvtb\" (UID: \"a7906cbf-9a94-4fe8-952d-48fed7a1cd3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9232e5-7b98c54cc4-ngvtb" Nov 03 21:15:46.545123 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.537597 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm7rn\" (UniqueName: \"kubernetes.io/projected/a7906cbf-9a94-4fe8-952d-48fed7a1cd3e-kube-api-access-mm7rn\") pod \"payload-processing-e2e-ipp-b-9232e5-7b98c54cc4-ngvtb\" (UID: \"a7906cbf-9a94-4fe8-952d-48fed7a1cd3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9232e5-7b98c54cc4-ngvtb" Nov 03 21:15:46.545123 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.539330 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7906cbf-9a94-4fe8-952d-48fed7a1cd3e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9232e5-7b98c54cc4-ngvtb\" (UID: \"a7906cbf-9a94-4fe8-952d-48fed7a1cd3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9232e5-7b98c54cc4-ngvtb" Nov 03 21:15:46.626673 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.626631 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm7rn\" (UniqueName: \"kubernetes.io/projected/a7906cbf-9a94-4fe8-952d-48fed7a1cd3e-kube-api-access-mm7rn\") pod \"payload-processing-e2e-ipp-b-9232e5-7b98c54cc4-ngvtb\" (UID: \"a7906cbf-9a94-4fe8-952d-48fed7a1cd3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9232e5-7b98c54cc4-ngvtb" Nov 03 21:15:46.673330 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:46.673097 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9232e5-7b98c54cc4-ngvtb" Nov 03 21:15:47.080845 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:47.080814 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9232e5-7b98c54cc4-ngvtb"] Nov 03 21:15:47.082567 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:15:47.082537 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7906cbf_9a94_4fe8_952d_48fed7a1cd3e.slice/crio-a104166eb7d6f3f0bcbdf30341fd284f099fa1da04120ebdfa0aa8dc0abf2c40 WatchSource:0}: Error finding container a104166eb7d6f3f0bcbdf30341fd284f099fa1da04120ebdfa0aa8dc0abf2c40: Status 404 returned error can't find the container with id a104166eb7d6f3f0bcbdf30341fd284f099fa1da04120ebdfa0aa8dc0abf2c40 Nov 03 21:15:47.083922 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:47.083897 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7"] Nov 03 21:15:47.117696 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:15:47.117658 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d863b8_3685_4948_b14b_9da2377616ff.slice/crio-a95adddf1982cc7ea521107bd336d7495ab7c3bfbd4fe7bf74f421ed3edba25d WatchSource:0}: Error finding container a95adddf1982cc7ea521107bd336d7495ab7c3bfbd4fe7bf74f421ed3edba25d: Status 404 returned error can't find the container with id a95adddf1982cc7ea521107bd336d7495ab7c3bfbd4fe7bf74f421ed3edba25d Nov 03 21:15:47.420481 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:47.420386 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7" event={"ID":"c1d863b8-3685-4948-b14b-9da2377616ff","Type":"ContainerStarted","Data":"a95adddf1982cc7ea521107bd336d7495ab7c3bfbd4fe7bf74f421ed3edba25d"} Nov 03 21:15:47.421989 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:47.421956 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9232e5-7b98c54cc4-ngvtb" event={"ID":"a7906cbf-9a94-4fe8-952d-48fed7a1cd3e","Type":"ContainerStarted","Data":"c71e46f314e497f17dec4ac0d4454b6081de0ab7246b2c64bf4b0c56658b5d2b"} Nov 03 21:15:47.422135 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:47.422010 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9232e5-7b98c54cc4-ngvtb" event={"ID":"a7906cbf-9a94-4fe8-952d-48fed7a1cd3e","Type":"ContainerStarted","Data":"a104166eb7d6f3f0bcbdf30341fd284f099fa1da04120ebdfa0aa8dc0abf2c40"} Nov 03 21:15:47.422201 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:47.422181 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9232e5-7b98c54cc4-ngvtb" Nov 03 21:15:47.533890 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:47.533834 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-9232e5-7b98c54cc4-ngvtb" podStartSLOduration=1.533819807 podStartE2EDuration="1.533819807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:47.531931754 +0000 UTC m=+1641.999710478" watchObservedRunningTime="2026-11-03 21:15:47.533819807 +0000 UTC m=+1642.001598523" Nov 03 21:15:48.429499 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:48.429460 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7" event={"ID":"c1d863b8-3685-4948-b14b-9da2377616ff","Type":"ContainerStarted","Data":"92f01a5dfb060587f3908c2e04655cbfba637cd15cd8de2fc36ac1a1f4f7f6d9"} Nov 03 21:15:48.429947 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:48.429562 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7" Nov 03 21:15:48.544638 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:48.544576 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7" podStartSLOduration=1.693340485 podStartE2EDuration="2.544559421s" totalImagesPullingTime="851.218936ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:48.542063748 +0000 UTC m=+1643.009842471" watchObservedRunningTime="2026-11-03 21:15:48.544559421 +0000 UTC m=+1643.012338143" Nov 03 21:15:50.674275 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:50.674236 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p"] Nov 03 21:15:50.674667 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:50.674473 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" podUID="5ba344bf-b7b1-4bfe-8098-78073ecc5fb6" containerName="istio-proxy" containerID="cri-o://64907aa41b720db14fffe3e1b295a757c90b094c329d04a21a3f025096ed5f9d" gracePeriod=30 Nov 03 21:15:54.441932 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:54.441896 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7" Nov 03 21:15:56.007112 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.006765 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:15:56.132096 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.129753 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istio-podinfo\") pod \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " Nov 03 21:15:56.132096 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.129934 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istio-data\") pod \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " Nov 03 21:15:56.132096 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.130016 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istiod-ca-cert\") pod \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " Nov 03 21:15:56.132096 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.130067 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-kube-api-access-r9w47\" (UniqueName: \"kubernetes.io/projected/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-kube-api-access-r9w47\") pod \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " Nov 03 21:15:56.132096 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.130147 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istio-envoy\") pod \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " Nov 03 21:15:56.132096 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.130178 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-workload-certs\") pod \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " Nov 03 21:15:56.132096 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.130204 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-workload-socket\") pod \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " Nov 03 21:15:56.132096 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.130258 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-credential-socket\") pod \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " Nov 03 21:15:56.132096 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.130307 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istio-token\" (UniqueName: \"kubernetes.io/projected/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istio-token\") pod \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\" (UID: \"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6\") " Nov 03 21:15:56.132096 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.131903 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istio-data" pod "5ba344bf-b7b1-4bfe-8098-78073ecc5fb6" (UID: "5ba344bf-b7b1-4bfe-8098-78073ecc5fb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:56.132096 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.131940 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-workload-certs" pod "5ba344bf-b7b1-4bfe-8098-78073ecc5fb6" (UID: "5ba344bf-b7b1-4bfe-8098-78073ecc5fb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:56.133334 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.133060 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-workload-socket" pod "5ba344bf-b7b1-4bfe-8098-78073ecc5fb6" (UID: "5ba344bf-b7b1-4bfe-8098-78073ecc5fb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:56.133334 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.133281 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-credential-socket" pod "5ba344bf-b7b1-4bfe-8098-78073ecc5fb6" (UID: "5ba344bf-b7b1-4bfe-8098-78073ecc5fb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:56.133512 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.133469 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istiod-ca-cert" pod "5ba344bf-b7b1-4bfe-8098-78073ecc5fb6" (UID: "5ba344bf-b7b1-4bfe-8098-78073ecc5fb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:56.134634 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.134610 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istio-envoy" pod "5ba344bf-b7b1-4bfe-8098-78073ecc5fb6" (UID: "5ba344bf-b7b1-4bfe-8098-78073ecc5fb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:56.138725 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.138674 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istio-podinfo" pod "5ba344bf-b7b1-4bfe-8098-78073ecc5fb6" (UID: "5ba344bf-b7b1-4bfe-8098-78073ecc5fb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:15:56.142581 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.142548 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istio-token" pod "5ba344bf-b7b1-4bfe-8098-78073ecc5fb6" (UID: "5ba344bf-b7b1-4bfe-8098-78073ecc5fb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:56.144628 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.144585 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-kube-api-access-r9w47" pod "5ba344bf-b7b1-4bfe-8098-78073ecc5fb6" (UID: "5ba344bf-b7b1-4bfe-8098-78073ecc5fb6"). InnerVolumeSpecName "kube-api-access-r9w47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:56.231880 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.231843 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:56.231880 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.231874 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:56.231880 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.231883 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:56.232166 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.231894 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9w47\" (UniqueName: \"kubernetes.io/projected/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-kube-api-access-r9w47\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:56.232166 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.231905 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:56.232166 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.231913 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:56.232166 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.231922 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:56.232166 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.231930 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:56.232166 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.231938 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:56.534102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.529773 5615 generic.go:356] "Generic (PLEG): container finished" podID="5ba344bf-b7b1-4bfe-8098-78073ecc5fb6" containerID="64907aa41b720db14fffe3e1b295a757c90b094c329d04a21a3f025096ed5f9d" exitCode=0 Nov 03 21:15:56.534102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.529866 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" event={"ID":"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6","Type":"ContainerDied","Data":"64907aa41b720db14fffe3e1b295a757c90b094c329d04a21a3f025096ed5f9d"} Nov 03 21:15:56.534102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.529927 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" event={"ID":"5ba344bf-b7b1-4bfe-8098-78073ecc5fb6","Type":"ContainerDied","Data":"712d68890f262662e4ceca39d33453c56c75ef0c463640724113869a6a6552b8"} Nov 03 21:15:56.534102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.529960 5615 scope.go:122] "RemoveContainer" containerID="64907aa41b720db14fffe3e1b295a757c90b094c329d04a21a3f025096ed5f9d" Nov 03 21:15:56.534102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.530179 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p" Nov 03 21:15:56.614331 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.607731 5615 scope.go:122] "RemoveContainer" containerID="499d514c2ee1c87367e85a5b68ecf0f9c459b88c03fc834bc7b1265a8cd3ad79" Nov 03 21:15:56.635968 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.635938 5615 scope.go:122] "RemoveContainer" containerID="64907aa41b720db14fffe3e1b295a757c90b094c329d04a21a3f025096ed5f9d" Nov 03 21:15:56.636581 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:15:56.636548 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64907aa41b720db14fffe3e1b295a757c90b094c329d04a21a3f025096ed5f9d\": container with ID starting with 64907aa41b720db14fffe3e1b295a757c90b094c329d04a21a3f025096ed5f9d not found: ID does not exist" containerID="64907aa41b720db14fffe3e1b295a757c90b094c329d04a21a3f025096ed5f9d" Nov 03 21:15:56.636734 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.636595 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64907aa41b720db14fffe3e1b295a757c90b094c329d04a21a3f025096ed5f9d"} err="failed to get container status \"64907aa41b720db14fffe3e1b295a757c90b094c329d04a21a3f025096ed5f9d\": rpc error: code = NotFound desc = could not find container \"64907aa41b720db14fffe3e1b295a757c90b094c329d04a21a3f025096ed5f9d\": container with ID starting with 64907aa41b720db14fffe3e1b295a757c90b094c329d04a21a3f025096ed5f9d not found: ID does not exist" Nov 03 21:15:56.636734 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.636636 5615 scope.go:122] "RemoveContainer" containerID="499d514c2ee1c87367e85a5b68ecf0f9c459b88c03fc834bc7b1265a8cd3ad79" Nov 03 21:15:56.636980 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:15:56.636958 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"499d514c2ee1c87367e85a5b68ecf0f9c459b88c03fc834bc7b1265a8cd3ad79\": container with ID starting with 499d514c2ee1c87367e85a5b68ecf0f9c459b88c03fc834bc7b1265a8cd3ad79 not found: ID does not exist" containerID="499d514c2ee1c87367e85a5b68ecf0f9c459b88c03fc834bc7b1265a8cd3ad79" Nov 03 21:15:56.637062 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.636983 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"499d514c2ee1c87367e85a5b68ecf0f9c459b88c03fc834bc7b1265a8cd3ad79"} err="failed to get container status \"499d514c2ee1c87367e85a5b68ecf0f9c459b88c03fc834bc7b1265a8cd3ad79\": rpc error: code = NotFound desc = could not find container \"499d514c2ee1c87367e85a5b68ecf0f9c459b88c03fc834bc7b1265a8cd3ad79\": container with ID starting with 499d514c2ee1c87367e85a5b68ecf0f9c459b88c03fc834bc7b1265a8cd3ad79 not found: ID does not exist" Nov 03 21:15:56.701678 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.701647 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p"] Nov 03 21:15:56.748114 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:56.748045 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-3e92e3-openshift-default-84f74fc947-fcv2p"] Nov 03 21:15:57.652689 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:57.652656 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ba344bf-b7b1-4bfe-8098-78073ecc5fb6" path="/var/lib/kubelet/pods/5ba344bf-b7b1-4bfe-8098-78073ecc5fb6/volumes" Nov 03 21:15:58.433892 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:58.433818 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9232e5-7b98c54cc4-ngvtb" Nov 03 21:15:58.534005 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:15:58.533968 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-22600c-68c974ccf4-kp5mr"] Nov 03 21:16:03.534780 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:03.534724 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-22600c-68c974ccf4-kp5mr" podUID="2858cbf3-8624-4362-8997-e62ae6013b53" containerName="payload-processing" containerID="cri-o://43d93e4a7e8fa171047b1062c3f372e2b5d0923dfaa14533727f4ff82d906409" gracePeriod=25 Nov 03 21:16:03.815771 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:03.815745 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-22600c-68c974ccf4-kp5mr" Nov 03 21:16:03.900489 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:03.900453 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2858cbf3-8624-4362-8997-e62ae6013b53-kube-api-access-jz7fj\" (UniqueName: \"kubernetes.io/projected/2858cbf3-8624-4362-8997-e62ae6013b53-kube-api-access-jz7fj\") pod \"2858cbf3-8624-4362-8997-e62ae6013b53\" (UID: \"2858cbf3-8624-4362-8997-e62ae6013b53\") " Nov 03 21:16:03.900489 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:03.900504 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2858cbf3-8624-4362-8997-e62ae6013b53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2858cbf3-8624-4362-8997-e62ae6013b53-plugins-config-volume\") pod \"2858cbf3-8624-4362-8997-e62ae6013b53\" (UID: \"2858cbf3-8624-4362-8997-e62ae6013b53\") " Nov 03 21:16:03.900956 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:03.900931 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2858cbf3-8624-4362-8997-e62ae6013b53-plugins-config-volume" pod "2858cbf3-8624-4362-8997-e62ae6013b53" (UID: "2858cbf3-8624-4362-8997-e62ae6013b53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:03.901198 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:03.901177 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2858cbf3-8624-4362-8997-e62ae6013b53-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:03.902709 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:03.902682 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2858cbf3-8624-4362-8997-e62ae6013b53-kube-api-access-jz7fj" pod "2858cbf3-8624-4362-8997-e62ae6013b53" (UID: "2858cbf3-8624-4362-8997-e62ae6013b53"). InnerVolumeSpecName "kube-api-access-jz7fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:04.002477 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:04.002439 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz7fj\" (UniqueName: \"kubernetes.io/projected/2858cbf3-8624-4362-8997-e62ae6013b53-kube-api-access-jz7fj\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:04.592448 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:04.592404 5615 generic.go:356] "Generic (PLEG): container finished" podID="2858cbf3-8624-4362-8997-e62ae6013b53" containerID="43d93e4a7e8fa171047b1062c3f372e2b5d0923dfaa14533727f4ff82d906409" exitCode=0 Nov 03 21:16:04.592954 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:04.592453 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-22600c-68c974ccf4-kp5mr" Nov 03 21:16:04.592954 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:04.592447 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-22600c-68c974ccf4-kp5mr" event={"ID":"2858cbf3-8624-4362-8997-e62ae6013b53","Type":"ContainerDied","Data":"43d93e4a7e8fa171047b1062c3f372e2b5d0923dfaa14533727f4ff82d906409"} Nov 03 21:16:04.592954 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:04.592601 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-22600c-68c974ccf4-kp5mr" event={"ID":"2858cbf3-8624-4362-8997-e62ae6013b53","Type":"ContainerDied","Data":"2c7243ebb2bbf84dff6b4187948763e414b2dc1f8639f4db8e85d260bdab0aca"} Nov 03 21:16:04.592954 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:04.592638 5615 scope.go:122] "RemoveContainer" containerID="43d93e4a7e8fa171047b1062c3f372e2b5d0923dfaa14533727f4ff82d906409" Nov 03 21:16:04.608202 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:04.608033 5615 scope.go:122] "RemoveContainer" containerID="43d93e4a7e8fa171047b1062c3f372e2b5d0923dfaa14533727f4ff82d906409" Nov 03 21:16:04.608627 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:16:04.608604 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43d93e4a7e8fa171047b1062c3f372e2b5d0923dfaa14533727f4ff82d906409\": container with ID starting with 43d93e4a7e8fa171047b1062c3f372e2b5d0923dfaa14533727f4ff82d906409 not found: ID does not exist" containerID="43d93e4a7e8fa171047b1062c3f372e2b5d0923dfaa14533727f4ff82d906409" Nov 03 21:16:04.608701 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:04.608641 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43d93e4a7e8fa171047b1062c3f372e2b5d0923dfaa14533727f4ff82d906409"} err="failed to get container status \"43d93e4a7e8fa171047b1062c3f372e2b5d0923dfaa14533727f4ff82d906409\": rpc error: code = NotFound desc = could not find container \"43d93e4a7e8fa171047b1062c3f372e2b5d0923dfaa14533727f4ff82d906409\": container with ID starting with 43d93e4a7e8fa171047b1062c3f372e2b5d0923dfaa14533727f4ff82d906409 not found: ID does not exist" Nov 03 21:16:04.749398 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:04.749362 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-22600c-68c974ccf4-kp5mr"] Nov 03 21:16:04.753339 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:04.753309 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-22600c-68c974ccf4-kp5mr"] Nov 03 21:16:05.659034 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:05.658995 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2858cbf3-8624-4362-8997-e62ae6013b53" path="/var/lib/kubelet/pods/2858cbf3-8624-4362-8997-e62ae6013b53/volumes" Nov 03 21:16:07.616329 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:07.616295 5615 generic.go:356] "Generic (PLEG): container finished" podID="26f4da92-abee-4727-90cc-e9f03dadaa12" containerID="3cf67f2f13c4a378830b88a229fe9a0d80b115e78e470e22613e19874b4df0fc" exitCode=0 Nov 03 21:16:07.616781 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:07.616370 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" event={"ID":"26f4da92-abee-4727-90cc-e9f03dadaa12","Type":"ContainerDied","Data":"3cf67f2f13c4a378830b88a229fe9a0d80b115e78e470e22613e19874b4df0fc"} Nov 03 21:16:07.616877 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:07.616858 5615 scope.go:122] "RemoveContainer" containerID="3cf67f2f13c4a378830b88a229fe9a0d80b115e78e470e22613e19874b4df0fc" Nov 03 21:16:07.618443 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:07.618408 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:16:07.618554 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:07.618498 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:16:07.618617 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:07.618549 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:16:08.629558 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:08.629512 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" event={"ID":"26f4da92-abee-4727-90cc-e9f03dadaa12","Type":"ContainerStarted","Data":"17c87f664a9084b69da98d7f242e5e174502807edca3e0e70eb9917414735f66"} Nov 03 21:16:09.132928 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:09.132887 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:16:09.134413 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:09.134387 5615 patch_prober.go:29] interesting pod/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 21:16:09.134539 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:09.134432 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" podUID="26f4da92-abee-4727-90cc-e9f03dadaa12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 21:16:09.940182 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:09.940143 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt"] Nov 03 21:16:09.940802 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:09.940788 5615 cpu_manager.go:414] "containerMap: removing container" podUID="5ba344bf-b7b1-4bfe-8098-78073ecc5fb6" containerName="istio-proxy" Nov 03 21:16:09.940845 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:09.940803 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba344bf-b7b1-4bfe-8098-78073ecc5fb6" containerName="istio-proxy" Nov 03 21:16:09.940845 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:09.940818 5615 cpu_manager.go:414] "containerMap: removing container" podUID="2858cbf3-8624-4362-8997-e62ae6013b53" containerName="payload-processing" Nov 03 21:16:09.940845 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:09.940823 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2858cbf3-8624-4362-8997-e62ae6013b53" containerName="payload-processing" Nov 03 21:16:09.940845 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:09.940842 5615 cpu_manager.go:414] "containerMap: removing container" podUID="5ba344bf-b7b1-4bfe-8098-78073ecc5fb6" containerName="istio-proxy" Nov 03 21:16:09.940970 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:09.940847 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba344bf-b7b1-4bfe-8098-78073ecc5fb6" containerName="istio-proxy" Nov 03 21:16:09.940970 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:09.940935 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba344bf-b7b1-4bfe-8098-78073ecc5fb6" containerName="istio-proxy" Nov 03 21:16:09.940970 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:09.940943 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="2858cbf3-8624-4362-8997-e62ae6013b53" containerName="payload-processing" Nov 03 21:16:09.940970 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:09.940953 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba344bf-b7b1-4bfe-8098-78073ecc5fb6" containerName="istio-proxy" Nov 03 21:16:09.943764 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:09.943743 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt" Nov 03 21:16:09.996303 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:09.996260 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-66513c-serving-cert\"" Nov 03 21:16:10.044058 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:10.044018 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt"] Nov 03 21:16:10.124031 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:10.123994 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfacf7de-fac7-47ee-84b7-4fe5a51e0b26-metrics-certs\") pod \"maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt\" (UID: \"dfacf7de-fac7-47ee-84b7-4fe5a51e0b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt" Nov 03 21:16:10.124031 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:10.124041 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll552\" (UniqueName: \"kubernetes.io/projected/dfacf7de-fac7-47ee-84b7-4fe5a51e0b26-kube-api-access-ll552\") pod \"maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt\" (UID: \"dfacf7de-fac7-47ee-84b7-4fe5a51e0b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt" Nov 03 21:16:10.124317 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:10.124270 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfacf7de-fac7-47ee-84b7-4fe5a51e0b26-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt\" (UID: \"dfacf7de-fac7-47ee-84b7-4fe5a51e0b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt" Nov 03 21:16:10.124375 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:10.124347 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfacf7de-fac7-47ee-84b7-4fe5a51e0b26-maas-api-tls\") pod \"maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt\" (UID: \"dfacf7de-fac7-47ee-84b7-4fe5a51e0b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt" Nov 03 21:16:10.138536 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:10.138493 5615 patch_prober.go:29] interesting pod/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 21:16:10.138748 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:10.138652 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" podUID="26f4da92-abee-4727-90cc-e9f03dadaa12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 21:16:10.230154 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:10.229587 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfacf7de-fac7-47ee-84b7-4fe5a51e0b26-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt\" (UID: \"dfacf7de-fac7-47ee-84b7-4fe5a51e0b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt" Nov 03 21:16:10.230154 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:10.229706 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfacf7de-fac7-47ee-84b7-4fe5a51e0b26-maas-api-tls\") pod \"maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt\" (UID: \"dfacf7de-fac7-47ee-84b7-4fe5a51e0b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt" Nov 03 21:16:10.230154 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:10.229811 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfacf7de-fac7-47ee-84b7-4fe5a51e0b26-metrics-certs\") pod \"maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt\" (UID: \"dfacf7de-fac7-47ee-84b7-4fe5a51e0b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt" Nov 03 21:16:10.230154 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:10.229836 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll552\" (UniqueName: \"kubernetes.io/projected/dfacf7de-fac7-47ee-84b7-4fe5a51e0b26-kube-api-access-ll552\") pod \"maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt\" (UID: \"dfacf7de-fac7-47ee-84b7-4fe5a51e0b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt" Nov 03 21:16:10.275873 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:10.274918 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfacf7de-fac7-47ee-84b7-4fe5a51e0b26-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt\" (UID: \"dfacf7de-fac7-47ee-84b7-4fe5a51e0b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt" Nov 03 21:16:10.293160 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:10.293113 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfacf7de-fac7-47ee-84b7-4fe5a51e0b26-metrics-certs\") pod \"maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt\" (UID: \"dfacf7de-fac7-47ee-84b7-4fe5a51e0b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt" Nov 03 21:16:10.294408 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:10.294346 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll552\" (UniqueName: \"kubernetes.io/projected/dfacf7de-fac7-47ee-84b7-4fe5a51e0b26-kube-api-access-ll552\") pod \"maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt\" (UID: \"dfacf7de-fac7-47ee-84b7-4fe5a51e0b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt" Nov 03 21:16:10.312098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:10.297767 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfacf7de-fac7-47ee-84b7-4fe5a51e0b26-maas-api-tls\") pod \"maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt\" (UID: \"dfacf7de-fac7-47ee-84b7-4fe5a51e0b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt" Nov 03 21:16:10.557634 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:10.557595 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt" Nov 03 21:16:10.624316 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:10.624186 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:16:10.653999 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:16:10.653957 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:16:10.956648 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:10.956605 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt"] Nov 03 21:16:10.957451 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:16:10.957423 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfacf7de_fac7_47ee_84b7_4fe5a51e0b26.slice/crio-8bb07615042f76ab3d3f65affda10a1a0f54826398a90f5b3415ccfff5fa44f5 WatchSource:0}: Error finding container 8bb07615042f76ab3d3f65affda10a1a0f54826398a90f5b3415ccfff5fa44f5: Status 404 returned error can't find the container with id 8bb07615042f76ab3d3f65affda10a1a0f54826398a90f5b3415ccfff5fa44f5 Nov 03 21:16:11.132766 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:11.132725 5615 patch_prober.go:29] interesting pod/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 21:16:11.132970 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:11.132795 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" podUID="26f4da92-abee-4727-90cc-e9f03dadaa12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 21:16:11.739718 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:11.739680 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt" event={"ID":"dfacf7de-fac7-47ee-84b7-4fe5a51e0b26","Type":"ContainerStarted","Data":"8bb07615042f76ab3d3f65affda10a1a0f54826398a90f5b3415ccfff5fa44f5"} Nov 03 21:16:12.133529 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:12.132368 5615 patch_prober.go:29] interesting pod/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 21:16:12.133529 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:12.132420 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" podUID="26f4da92-abee-4727-90cc-e9f03dadaa12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 21:16:12.747589 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:12.747552 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt" event={"ID":"dfacf7de-fac7-47ee-84b7-4fe5a51e0b26","Type":"ContainerStarted","Data":"da0724ab87045a3202c7b2267d7e0a78c187939893f8ed35ecceff2fa831f41a"} Nov 03 21:16:12.747792 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:12.747610 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt" Nov 03 21:16:12.861877 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:12.861826 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt" podStartSLOduration=3.024494545 podStartE2EDuration="3.861810191s" totalImagesPullingTime="837.315646ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:16:12.859433193 +0000 UTC m=+1667.327211930" watchObservedRunningTime="2026-11-03 21:16:12.861810191 +0000 UTC m=+1667.329588913" Nov 03 21:16:13.145194 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:13.137941 5615 patch_prober.go:29] interesting pod/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 21:16:13.145194 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:13.138017 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" podUID="26f4da92-abee-4727-90cc-e9f03dadaa12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 21:16:14.136948 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:14.136917 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:16:14.137955 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:14.137923 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:16:18.761469 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:18.761437 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt" Nov 03 21:16:24.931339 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:24.931302 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-78b68f9566-6hnx6"] Nov 03 21:16:24.934988 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:24.934962 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-78b68f9566-6hnx6" Nov 03 21:16:24.991587 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:24.991551 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-78b68f9566-6hnx6"] Nov 03 21:16:25.008207 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:25.008167 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkcj5\" (UniqueName: \"kubernetes.io/projected/95aee87a-8056-47a5-9574-02d25a20f331-kube-api-access-qkcj5\") pod \"payload-pre-processing-e2e-ipp-a-0f8726-78b68f9566-6hnx6\" (UID: \"95aee87a-8056-47a5-9574-02d25a20f331\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-78b68f9566-6hnx6" Nov 03 21:16:25.008456 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:25.008430 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95aee87a-8056-47a5-9574-02d25a20f331-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0f8726-78b68f9566-6hnx6\" (UID: \"95aee87a-8056-47a5-9574-02d25a20f331\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-78b68f9566-6hnx6" Nov 03 21:16:25.035600 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:25.035557 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0f8726\"" Nov 03 21:16:25.035600 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:25.035601 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0f8726-dockercfg-zkhlj\"" Nov 03 21:16:25.120485 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:25.120437 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkcj5\" (UniqueName: \"kubernetes.io/projected/95aee87a-8056-47a5-9574-02d25a20f331-kube-api-access-qkcj5\") pod \"payload-pre-processing-e2e-ipp-a-0f8726-78b68f9566-6hnx6\" (UID: \"95aee87a-8056-47a5-9574-02d25a20f331\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-78b68f9566-6hnx6" Nov 03 21:16:25.120645 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:25.120624 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95aee87a-8056-47a5-9574-02d25a20f331-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0f8726-78b68f9566-6hnx6\" (UID: \"95aee87a-8056-47a5-9574-02d25a20f331\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-78b68f9566-6hnx6" Nov 03 21:16:25.121457 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:25.121428 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95aee87a-8056-47a5-9574-02d25a20f331-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0f8726-78b68f9566-6hnx6\" (UID: \"95aee87a-8056-47a5-9574-02d25a20f331\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-78b68f9566-6hnx6" Nov 03 21:16:25.178861 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:25.178820 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkcj5\" (UniqueName: \"kubernetes.io/projected/95aee87a-8056-47a5-9574-02d25a20f331-kube-api-access-qkcj5\") pod \"payload-pre-processing-e2e-ipp-a-0f8726-78b68f9566-6hnx6\" (UID: \"95aee87a-8056-47a5-9574-02d25a20f331\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-78b68f9566-6hnx6" Nov 03 21:16:25.250042 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:25.250001 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-78b68f9566-6hnx6" Nov 03 21:16:25.628278 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:25.628252 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-78b68f9566-6hnx6"] Nov 03 21:16:25.629919 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:16:25.629894 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95aee87a_8056_47a5_9574_02d25a20f331.slice/crio-d5f111a5a11dff8773b476fb5985cb980e5102359c90d211bf779c1c5e739b11 WatchSource:0}: Error finding container d5f111a5a11dff8773b476fb5985cb980e5102359c90d211bf779c1c5e739b11: Status 404 returned error can't find the container with id d5f111a5a11dff8773b476fb5985cb980e5102359c90d211bf779c1c5e739b11 Nov 03 21:16:25.834941 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:25.834904 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-78b68f9566-6hnx6" event={"ID":"95aee87a-8056-47a5-9574-02d25a20f331","Type":"ContainerStarted","Data":"d5f111a5a11dff8773b476fb5985cb980e5102359c90d211bf779c1c5e739b11"} Nov 03 21:16:27.852603 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:27.852547 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-78b68f9566-6hnx6" event={"ID":"95aee87a-8056-47a5-9574-02d25a20f331","Type":"ContainerStarted","Data":"02e2a94d8703d085d7951fd65c8539de2e9a80a69b928c32c347318d48b355fc"} Nov 03 21:16:27.853157 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:27.852844 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-78b68f9566-6hnx6" Nov 03 21:16:27.966880 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:27.966820 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-78b68f9566-6hnx6" podStartSLOduration=2.298084283 podStartE2EDuration="3.966805357s" totalImagesPullingTime="1.668721074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:16:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:16:27.963755296 +0000 UTC m=+1682.431534020" watchObservedRunningTime="2026-11-03 21:16:27.966805357 +0000 UTC m=+1682.434584111" Nov 03 21:16:38.860970 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:38.860937 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-78b68f9566-6hnx6" Nov 03 21:16:39.465599 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:39.465565 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k"] Nov 03 21:16:39.469262 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:39.469240 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k" Nov 03 21:16:39.501606 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:39.501562 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4539a85b-3f5b-40e1-9caa-9407bf616dc8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k\" (UID: \"4539a85b-3f5b-40e1-9caa-9407bf616dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k" Nov 03 21:16:39.501781 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:39.501736 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4539a85b-3f5b-40e1-9caa-9407bf616dc8-metrics-certs\") pod \"maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k\" (UID: \"4539a85b-3f5b-40e1-9caa-9407bf616dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k" Nov 03 21:16:39.501781 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:39.501777 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4539a85b-3f5b-40e1-9caa-9407bf616dc8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k\" (UID: \"4539a85b-3f5b-40e1-9caa-9407bf616dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k" Nov 03 21:16:39.501888 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:39.501873 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq4lk\" (UniqueName: \"kubernetes.io/projected/4539a85b-3f5b-40e1-9caa-9407bf616dc8-kube-api-access-bq4lk\") pod \"maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k\" (UID: \"4539a85b-3f5b-40e1-9caa-9407bf616dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k" Nov 03 21:16:39.522725 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:39.522694 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-9232e5-serving-cert\"" Nov 03 21:16:39.526958 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:39.526924 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k"] Nov 03 21:16:39.602852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:39.602812 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4539a85b-3f5b-40e1-9caa-9407bf616dc8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k\" (UID: \"4539a85b-3f5b-40e1-9caa-9407bf616dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k" Nov 03 21:16:39.603042 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:39.602922 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4539a85b-3f5b-40e1-9caa-9407bf616dc8-metrics-certs\") pod \"maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k\" (UID: \"4539a85b-3f5b-40e1-9caa-9407bf616dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k" Nov 03 21:16:39.603042 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:39.602951 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4539a85b-3f5b-40e1-9caa-9407bf616dc8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k\" (UID: \"4539a85b-3f5b-40e1-9caa-9407bf616dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k" Nov 03 21:16:39.603042 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:39.603002 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq4lk\" (UniqueName: \"kubernetes.io/projected/4539a85b-3f5b-40e1-9caa-9407bf616dc8-kube-api-access-bq4lk\") pod \"maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k\" (UID: \"4539a85b-3f5b-40e1-9caa-9407bf616dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k" Nov 03 21:16:39.603560 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:39.603535 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4539a85b-3f5b-40e1-9caa-9407bf616dc8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k\" (UID: \"4539a85b-3f5b-40e1-9caa-9407bf616dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k" Nov 03 21:16:39.603665 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:16:39.603639 5615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-serving-cert: secret "maas-api-e2e-ipp-b-9232e5-serving-cert" not found Nov 03 21:16:39.603724 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:16:39.603704 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4539a85b-3f5b-40e1-9caa-9407bf616dc8-maas-api-tls podName:4539a85b-3f5b-40e1-9caa-9407bf616dc8 nodeName:}" failed. No retries permitted until 2026-11-03 21:16:40.103684978 +0000 UTC m=+1694.571463679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4539a85b-3f5b-40e1-9caa-9407bf616dc8-maas-api-tls") pod "maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k" (UID: "4539a85b-3f5b-40e1-9caa-9407bf616dc8") : secret "maas-api-e2e-ipp-b-9232e5-serving-cert" not found Nov 03 21:16:39.605783 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:39.605758 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4539a85b-3f5b-40e1-9caa-9407bf616dc8-metrics-certs\") pod \"maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k\" (UID: \"4539a85b-3f5b-40e1-9caa-9407bf616dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k" Nov 03 21:16:39.666067 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:39.666035 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq4lk\" (UniqueName: \"kubernetes.io/projected/4539a85b-3f5b-40e1-9caa-9407bf616dc8-kube-api-access-bq4lk\") pod \"maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k\" (UID: \"4539a85b-3f5b-40e1-9caa-9407bf616dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k" Nov 03 21:16:40.132118 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:40.132060 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4539a85b-3f5b-40e1-9caa-9407bf616dc8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k\" (UID: \"4539a85b-3f5b-40e1-9caa-9407bf616dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k" Nov 03 21:16:40.134566 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:40.134542 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4539a85b-3f5b-40e1-9caa-9407bf616dc8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k\" (UID: \"4539a85b-3f5b-40e1-9caa-9407bf616dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k" Nov 03 21:16:40.382847 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:40.382760 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k" Nov 03 21:16:40.619425 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:16:40.619385 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:16:40.759358 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:40.759328 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k"] Nov 03 21:16:40.761277 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:16:40.761246 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4539a85b_3f5b_40e1_9caa_9407bf616dc8.slice/crio-be3518b69e7cfec249faa1286a118d78fa9a55148a482780a0463877525fb822 WatchSource:0}: Error finding container be3518b69e7cfec249faa1286a118d78fa9a55148a482780a0463877525fb822: Status 404 returned error can't find the container with id be3518b69e7cfec249faa1286a118d78fa9a55148a482780a0463877525fb822 Nov 03 21:16:40.985688 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:40.985653 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k" event={"ID":"4539a85b-3f5b-40e1-9caa-9407bf616dc8","Type":"ContainerStarted","Data":"be3518b69e7cfec249faa1286a118d78fa9a55148a482780a0463877525fb822"} Nov 03 21:16:41.994109 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:41.994058 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k" event={"ID":"4539a85b-3f5b-40e1-9caa-9407bf616dc8","Type":"ContainerStarted","Data":"2a36ca7d8124bb12bda5b8dbe505174f4caef277afc7f5b17773112434b8bfbb"} Nov 03 21:16:41.994573 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:41.994156 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k" Nov 03 21:16:42.108777 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:42.108665 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k" podStartSLOduration=2.109322582 podStartE2EDuration="3.108646148s" totalImagesPullingTime="999.323566ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:16:42.106731954 +0000 UTC m=+1696.574510677" watchObservedRunningTime="2026-11-03 21:16:42.108646148 +0000 UTC m=+1696.576424877" Nov 03 21:16:48.005726 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:48.005697 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k" Nov 03 21:16:50.787298 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:50.787257 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-812243-75947bbfb8-lnlqk"] Nov 03 21:16:53.035015 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:53.034966 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz"] Nov 03 21:16:53.035465 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:53.035273 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz" podUID="86bccd8d-d5c7-4e37-8a69-334f7b1bdd48" containerName="maas-api" containerID="cri-o://df9b88913f3d02fdfa61bd7c292e2e8441f0747c4ab0e0c528e763f8886c608f" gracePeriod=30 Nov 03 21:16:53.038743 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:53.038709 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz" podUID="86bccd8d-d5c7-4e37-8a69-334f7b1bdd48" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.95:8443/health\": dial tcp 10.134.0.95:8443: connect: connection refused" Nov 03 21:16:53.409265 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:53.409220 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz" Nov 03 21:16:53.596772 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:53.596681 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-openshift-service-ca\") pod \"86bccd8d-d5c7-4e37-8a69-334f7b1bdd48\" (UID: \"86bccd8d-d5c7-4e37-8a69-334f7b1bdd48\") " Nov 03 21:16:53.597152 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:53.597129 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-metrics-certs\" (UniqueName: \"kubernetes.io/secret/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-metrics-certs\") pod \"86bccd8d-d5c7-4e37-8a69-334f7b1bdd48\" (UID: \"86bccd8d-d5c7-4e37-8a69-334f7b1bdd48\") " Nov 03 21:16:53.597413 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:53.597397 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-kube-api-access-7zqgw\" (UniqueName: \"kubernetes.io/projected/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-kube-api-access-7zqgw\") pod \"86bccd8d-d5c7-4e37-8a69-334f7b1bdd48\" (UID: \"86bccd8d-d5c7-4e37-8a69-334f7b1bdd48\") " Nov 03 21:16:53.597582 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:53.597565 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-maas-api-tls\") pod \"86bccd8d-d5c7-4e37-8a69-334f7b1bdd48\" (UID: \"86bccd8d-d5c7-4e37-8a69-334f7b1bdd48\") " Nov 03 21:16:53.617732 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:53.617691 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-openshift-service-ca" pod "86bccd8d-d5c7-4e37-8a69-334f7b1bdd48" (UID: "86bccd8d-d5c7-4e37-8a69-334f7b1bdd48"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:53.621408 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:53.621371 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:53.629441 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:53.629402 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-maas-api-tls" pod "86bccd8d-d5c7-4e37-8a69-334f7b1bdd48" (UID: "86bccd8d-d5c7-4e37-8a69-334f7b1bdd48"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:16:53.630118 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:53.630060 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-kube-api-access-7zqgw" pod "86bccd8d-d5c7-4e37-8a69-334f7b1bdd48" (UID: "86bccd8d-d5c7-4e37-8a69-334f7b1bdd48"). InnerVolumeSpecName "kube-api-access-7zqgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:53.631843 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:53.630354 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-metrics-certs" pod "86bccd8d-d5c7-4e37-8a69-334f7b1bdd48" (UID: "86bccd8d-d5c7-4e37-8a69-334f7b1bdd48"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:16:53.733096 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:53.724262 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:53.733096 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:53.724308 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zqgw\" (UniqueName: \"kubernetes.io/projected/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-kube-api-access-7zqgw\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:53.733096 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:53.724325 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:54.194969 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:54.188920 5615 generic.go:356] "Generic (PLEG): container finished" podID="86bccd8d-d5c7-4e37-8a69-334f7b1bdd48" containerID="df9b88913f3d02fdfa61bd7c292e2e8441f0747c4ab0e0c528e763f8886c608f" exitCode=0 Nov 03 21:16:54.194969 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:54.189199 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz" event={"ID":"86bccd8d-d5c7-4e37-8a69-334f7b1bdd48","Type":"ContainerDied","Data":"df9b88913f3d02fdfa61bd7c292e2e8441f0747c4ab0e0c528e763f8886c608f"} Nov 03 21:16:54.194969 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:54.189254 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz" event={"ID":"86bccd8d-d5c7-4e37-8a69-334f7b1bdd48","Type":"ContainerDied","Data":"0e1a798475a745d40c0071f8b04d9da4c41a54f40cc2cb7422d375117230acc2"} Nov 03 21:16:54.194969 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:54.189283 5615 scope.go:122] "RemoveContainer" containerID="df9b88913f3d02fdfa61bd7c292e2e8441f0747c4ab0e0c528e763f8886c608f" Nov 03 21:16:54.194969 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:54.189461 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz" Nov 03 21:16:54.222965 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:54.222938 5615 scope.go:122] "RemoveContainer" containerID="df9b88913f3d02fdfa61bd7c292e2e8441f0747c4ab0e0c528e763f8886c608f" Nov 03 21:16:54.223390 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:16:54.223355 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df9b88913f3d02fdfa61bd7c292e2e8441f0747c4ab0e0c528e763f8886c608f\": container with ID starting with df9b88913f3d02fdfa61bd7c292e2e8441f0747c4ab0e0c528e763f8886c608f not found: ID does not exist" containerID="df9b88913f3d02fdfa61bd7c292e2e8441f0747c4ab0e0c528e763f8886c608f" Nov 03 21:16:54.223484 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:54.223412 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df9b88913f3d02fdfa61bd7c292e2e8441f0747c4ab0e0c528e763f8886c608f"} err="failed to get container status \"df9b88913f3d02fdfa61bd7c292e2e8441f0747c4ab0e0c528e763f8886c608f\": rpc error: code = NotFound desc = could not find container \"df9b88913f3d02fdfa61bd7c292e2e8441f0747c4ab0e0c528e763f8886c608f\": container with ID starting with df9b88913f3d02fdfa61bd7c292e2e8441f0747c4ab0e0c528e763f8886c608f not found: ID does not exist" Nov 03 21:16:54.347547 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:54.347510 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz"] Nov 03 21:16:54.354474 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:54.354441 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-22600c-796b7d6dd7-bwzsz"] Nov 03 21:16:55.653010 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:55.652978 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86bccd8d-d5c7-4e37-8a69-334f7b1bdd48" path="/var/lib/kubelet/pods/86bccd8d-d5c7-4e37-8a69-334f7b1bdd48/volumes" Nov 03 21:16:55.788033 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:55.787984 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-75947bbfb8-lnlqk" podUID="c06f0f6d-1bd4-4fcc-a68a-1080bda86136" containerName="payload-processing" containerID="cri-o://12cda3ae01a1a57326c1d855a1d418d458446af3c542c254754de7147108cb60" gracePeriod=25 Nov 03 21:16:56.075621 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:56.075585 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-75947bbfb8-lnlqk" Nov 03 21:16:56.162171 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:56.162088 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c06f0f6d-1bd4-4fcc-a68a-1080bda86136-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c06f0f6d-1bd4-4fcc-a68a-1080bda86136-plugins-config-volume\") pod \"c06f0f6d-1bd4-4fcc-a68a-1080bda86136\" (UID: \"c06f0f6d-1bd4-4fcc-a68a-1080bda86136\") " Nov 03 21:16:56.162171 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:56.162137 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c06f0f6d-1bd4-4fcc-a68a-1080bda86136-kube-api-access-v2dtt\" (UniqueName: \"kubernetes.io/projected/c06f0f6d-1bd4-4fcc-a68a-1080bda86136-kube-api-access-v2dtt\") pod \"c06f0f6d-1bd4-4fcc-a68a-1080bda86136\" (UID: \"c06f0f6d-1bd4-4fcc-a68a-1080bda86136\") " Nov 03 21:16:56.162474 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:56.162448 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c06f0f6d-1bd4-4fcc-a68a-1080bda86136-plugins-config-volume" pod "c06f0f6d-1bd4-4fcc-a68a-1080bda86136" (UID: "c06f0f6d-1bd4-4fcc-a68a-1080bda86136"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:56.163760 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:56.163720 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c06f0f6d-1bd4-4fcc-a68a-1080bda86136-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:56.170369 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:56.170318 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c06f0f6d-1bd4-4fcc-a68a-1080bda86136-kube-api-access-v2dtt" pod "c06f0f6d-1bd4-4fcc-a68a-1080bda86136" (UID: "c06f0f6d-1bd4-4fcc-a68a-1080bda86136"). InnerVolumeSpecName "kube-api-access-v2dtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:56.205271 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:56.205235 5615 generic.go:356] "Generic (PLEG): container finished" podID="c06f0f6d-1bd4-4fcc-a68a-1080bda86136" containerID="12cda3ae01a1a57326c1d855a1d418d458446af3c542c254754de7147108cb60" exitCode=0 Nov 03 21:16:56.205569 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:56.205309 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-75947bbfb8-lnlqk" event={"ID":"c06f0f6d-1bd4-4fcc-a68a-1080bda86136","Type":"ContainerDied","Data":"12cda3ae01a1a57326c1d855a1d418d458446af3c542c254754de7147108cb60"} Nov 03 21:16:56.205569 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:56.205314 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-75947bbfb8-lnlqk" Nov 03 21:16:56.205569 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:56.205378 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-75947bbfb8-lnlqk" event={"ID":"c06f0f6d-1bd4-4fcc-a68a-1080bda86136","Type":"ContainerDied","Data":"73634bfa7b6d84beafc695863d30a681827efbeb4d5517de93ad6c3347910672"} Nov 03 21:16:56.205569 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:56.205402 5615 scope.go:122] "RemoveContainer" containerID="12cda3ae01a1a57326c1d855a1d418d458446af3c542c254754de7147108cb60" Nov 03 21:16:56.216750 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:56.216724 5615 scope.go:122] "RemoveContainer" containerID="12cda3ae01a1a57326c1d855a1d418d458446af3c542c254754de7147108cb60" Nov 03 21:16:56.217068 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:16:56.217049 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12cda3ae01a1a57326c1d855a1d418d458446af3c542c254754de7147108cb60\": container with ID starting with 12cda3ae01a1a57326c1d855a1d418d458446af3c542c254754de7147108cb60 not found: ID does not exist" containerID="12cda3ae01a1a57326c1d855a1d418d458446af3c542c254754de7147108cb60" Nov 03 21:16:56.217170 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:56.217113 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12cda3ae01a1a57326c1d855a1d418d458446af3c542c254754de7147108cb60"} err="failed to get container status \"12cda3ae01a1a57326c1d855a1d418d458446af3c542c254754de7147108cb60\": rpc error: code = NotFound desc = could not find container \"12cda3ae01a1a57326c1d855a1d418d458446af3c542c254754de7147108cb60\": container with ID starting with 12cda3ae01a1a57326c1d855a1d418d458446af3c542c254754de7147108cb60 not found: ID does not exist" Nov 03 21:16:56.264742 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:56.264698 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2dtt\" (UniqueName: \"kubernetes.io/projected/c06f0f6d-1bd4-4fcc-a68a-1080bda86136-kube-api-access-v2dtt\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:56.361633 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:56.361598 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-812243-75947bbfb8-lnlqk"] Nov 03 21:16:56.366813 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:56.366782 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-812243-75947bbfb8-lnlqk"] Nov 03 21:16:57.654730 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:16:57.654698 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c06f0f6d-1bd4-4fcc-a68a-1080bda86136" path="/var/lib/kubelet/pods/c06f0f6d-1bd4-4fcc-a68a-1080bda86136/volumes" Nov 03 21:17:01.279243 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:01.278530 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-79d96df5c6-cfzcv"] Nov 03 21:17:01.289107 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:01.284707 5615 cpu_manager.go:414] "containerMap: removing container" podUID="86bccd8d-d5c7-4e37-8a69-334f7b1bdd48" containerName="maas-api" Nov 03 21:17:01.289107 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:01.284746 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86bccd8d-d5c7-4e37-8a69-334f7b1bdd48" containerName="maas-api" Nov 03 21:17:01.289107 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:01.284781 5615 cpu_manager.go:414] "containerMap: removing container" podUID="c06f0f6d-1bd4-4fcc-a68a-1080bda86136" containerName="payload-processing" Nov 03 21:17:01.289107 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:01.284791 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c06f0f6d-1bd4-4fcc-a68a-1080bda86136" containerName="payload-processing" Nov 03 21:17:01.289107 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:01.285045 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="86bccd8d-d5c7-4e37-8a69-334f7b1bdd48" containerName="maas-api" Nov 03 21:17:01.289107 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:01.285088 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="c06f0f6d-1bd4-4fcc-a68a-1080bda86136" containerName="payload-processing" Nov 03 21:17:01.299197 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:01.299165 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-79d96df5c6-cfzcv" Nov 03 21:17:01.351378 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:01.351346 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b2ad2d\"" Nov 03 21:17:01.366470 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:01.366426 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-79d96df5c6-cfzcv"] Nov 03 21:17:01.399100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:01.388933 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b2ad2d-dockercfg-kc2lp\"" Nov 03 21:17:01.429809 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:01.429768 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-b7cdc65f9-vsbx8"] Nov 03 21:17:01.435167 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:01.435126 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm4nh\" (UniqueName: \"kubernetes.io/projected/81109a63-4170-413c-bad7-4b49eaf27f0e-kube-api-access-cm4nh\") pod \"payload-processing-e2e-ipp-b-b2ad2d-79d96df5c6-cfzcv\" (UID: \"81109a63-4170-413c-bad7-4b49eaf27f0e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-79d96df5c6-cfzcv" Nov 03 21:17:01.435576 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:01.435554 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81109a63-4170-413c-bad7-4b49eaf27f0e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b2ad2d-79d96df5c6-cfzcv\" (UID: \"81109a63-4170-413c-bad7-4b49eaf27f0e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-79d96df5c6-cfzcv" Nov 03 21:17:01.443425 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:01.443389 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-b7cdc65f9-vsbx8" Nov 03 21:17:01.485852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:01.485819 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-b7cdc65f9-vsbx8"] Nov 03 21:17:01.536979 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:01.536893 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm4nh\" (UniqueName: \"kubernetes.io/projected/81109a63-4170-413c-bad7-4b49eaf27f0e-kube-api-access-cm4nh\") pod \"payload-processing-e2e-ipp-b-b2ad2d-79d96df5c6-cfzcv\" (UID: \"81109a63-4170-413c-bad7-4b49eaf27f0e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-79d96df5c6-cfzcv" Nov 03 21:17:01.537163 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:01.537039 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d82588a-f1d2-424e-b718-081993dc868c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b2ad2d-b7cdc65f9-vsbx8\" (UID: \"0d82588a-f1d2-424e-b718-081993dc868c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-b7cdc65f9-vsbx8" Nov 03 21:17:01.537219 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:01.537180 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81109a63-4170-413c-bad7-4b49eaf27f0e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b2ad2d-79d96df5c6-cfzcv\" (UID: \"81109a63-4170-413c-bad7-4b49eaf27f0e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-79d96df5c6-cfzcv" Nov 03 21:17:01.537270 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:01.537252 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m74h5\" (UniqueName: \"kubernetes.io/projected/0d82588a-f1d2-424e-b718-081993dc868c-kube-api-access-m74h5\") pod \"payload-pre-processing-e2e-ipp-b-b2ad2d-b7cdc65f9-vsbx8\" (UID: \"0d82588a-f1d2-424e-b718-081993dc868c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-b7cdc65f9-vsbx8" Nov 03 21:17:01.537836 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:01.537816 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81109a63-4170-413c-bad7-4b49eaf27f0e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b2ad2d-79d96df5c6-cfzcv\" (UID: \"81109a63-4170-413c-bad7-4b49eaf27f0e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-79d96df5c6-cfzcv" Nov 03 21:17:01.595062 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:01.595021 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm4nh\" (UniqueName: \"kubernetes.io/projected/81109a63-4170-413c-bad7-4b49eaf27f0e-kube-api-access-cm4nh\") pod \"payload-processing-e2e-ipp-b-b2ad2d-79d96df5c6-cfzcv\" (UID: \"81109a63-4170-413c-bad7-4b49eaf27f0e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-79d96df5c6-cfzcv" Nov 03 21:17:01.638682 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:01.638645 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d82588a-f1d2-424e-b718-081993dc868c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b2ad2d-b7cdc65f9-vsbx8\" (UID: \"0d82588a-f1d2-424e-b718-081993dc868c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-b7cdc65f9-vsbx8" Nov 03 21:17:01.638860 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:01.638757 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m74h5\" (UniqueName: \"kubernetes.io/projected/0d82588a-f1d2-424e-b718-081993dc868c-kube-api-access-m74h5\") pod \"payload-pre-processing-e2e-ipp-b-b2ad2d-b7cdc65f9-vsbx8\" (UID: \"0d82588a-f1d2-424e-b718-081993dc868c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-b7cdc65f9-vsbx8" Nov 03 21:17:01.639429 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:01.639406 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d82588a-f1d2-424e-b718-081993dc868c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b2ad2d-b7cdc65f9-vsbx8\" (UID: \"0d82588a-f1d2-424e-b718-081993dc868c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-b7cdc65f9-vsbx8" Nov 03 21:17:01.650721 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:01.650691 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-79d96df5c6-cfzcv" Nov 03 21:17:01.716107 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:01.697517 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m74h5\" (UniqueName: \"kubernetes.io/projected/0d82588a-f1d2-424e-b718-081993dc868c-kube-api-access-m74h5\") pod \"payload-pre-processing-e2e-ipp-b-b2ad2d-b7cdc65f9-vsbx8\" (UID: \"0d82588a-f1d2-424e-b718-081993dc868c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-b7cdc65f9-vsbx8" Nov 03 21:17:01.773363 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:01.773326 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-b7cdc65f9-vsbx8" Nov 03 21:17:02.054707 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:02.054672 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-79d96df5c6-cfzcv"] Nov 03 21:17:02.056137 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:17:02.056104 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81109a63_4170_413c_bad7_4b49eaf27f0e.slice/crio-44b2488f45b5bfd57abd37243d3106cb4394cf54be3444811b5634d2b1f4b41f WatchSource:0}: Error finding container 44b2488f45b5bfd57abd37243d3106cb4394cf54be3444811b5634d2b1f4b41f: Status 404 returned error can't find the container with id 44b2488f45b5bfd57abd37243d3106cb4394cf54be3444811b5634d2b1f4b41f Nov 03 21:17:02.183909 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:02.183864 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-b7cdc65f9-vsbx8"] Nov 03 21:17:02.186002 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:17:02.185950 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d82588a_f1d2_424e_b718_081993dc868c.slice/crio-d679b64ada038b02069c0db37e73fd08fdffc9db0aa70788c3fe9b68e0db7fc9 WatchSource:0}: Error finding container d679b64ada038b02069c0db37e73fd08fdffc9db0aa70788c3fe9b68e0db7fc9: Status 404 returned error can't find the container with id d679b64ada038b02069c0db37e73fd08fdffc9db0aa70788c3fe9b68e0db7fc9 Nov 03 21:17:02.329107 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:02.329046 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-79d96df5c6-cfzcv" event={"ID":"81109a63-4170-413c-bad7-4b49eaf27f0e","Type":"ContainerStarted","Data":"cbe3bbd50f786b7219a9e02dd984fca77dc829f678ad0aecf495a59935a67987"} Nov 03 21:17:02.329525 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:02.329502 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-79d96df5c6-cfzcv" event={"ID":"81109a63-4170-413c-bad7-4b49eaf27f0e","Type":"ContainerStarted","Data":"44b2488f45b5bfd57abd37243d3106cb4394cf54be3444811b5634d2b1f4b41f"} Nov 03 21:17:02.329648 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:02.329569 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-79d96df5c6-cfzcv" Nov 03 21:17:02.331629 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:02.331559 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-b7cdc65f9-vsbx8" event={"ID":"0d82588a-f1d2-424e-b718-081993dc868c","Type":"ContainerStarted","Data":"d679b64ada038b02069c0db37e73fd08fdffc9db0aa70788c3fe9b68e0db7fc9"} Nov 03 21:17:02.442125 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:02.442040 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-79d96df5c6-cfzcv" podStartSLOduration=1.442023376 podStartE2EDuration="1.442023376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:17:02.440991221 +0000 UTC m=+1716.908769944" watchObservedRunningTime="2026-11-03 21:17:02.442023376 +0000 UTC m=+1716.909802098" Nov 03 21:17:03.340337 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:03.340298 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-b7cdc65f9-vsbx8" event={"ID":"0d82588a-f1d2-424e-b718-081993dc868c","Type":"ContainerStarted","Data":"946698b96c6c8b5500066ca66c69663a50a0c7f8fbf10b2589f041e70e310267"} Nov 03 21:17:03.340843 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:03.340570 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-b7cdc65f9-vsbx8" Nov 03 21:17:03.450433 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:03.450377 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-b7cdc65f9-vsbx8" podStartSLOduration=2.4503622480000002 podStartE2EDuration="2.450362248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:17:03.449513329 +0000 UTC m=+1717.917292054" watchObservedRunningTime="2026-11-03 21:17:03.450362248 +0000 UTC m=+1717.918140971" Nov 03 21:17:06.001862 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:06.001821 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-gtbjq"] Nov 03 21:17:08.323977 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.323936 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7"] Nov 03 21:17:08.328303 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.328276 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:08.367742 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.367707 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-istio-data\") pod \"e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:08.367742 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.367745 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f82eafd0-9398-4cec-9b95-e510391864ab-istio-token\") pod \"e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:08.367993 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.367774 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f82eafd0-9398-4cec-9b95-e510391864ab-istiod-ca-cert\") pod \"e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:08.368087 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.368046 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f82eafd0-9398-4cec-9b95-e510391864ab-istio-podinfo\") pod \"e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:08.368178 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.368162 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-workload-certs\") pod \"e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:08.368278 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.368264 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-credential-socket\") pod \"e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:08.368340 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.368297 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-workload-socket\") pod \"e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:08.368410 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.368395 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-istio-envoy\") pod \"e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:08.368459 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.368426 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r98pf\" (UniqueName: \"kubernetes.io/projected/f82eafd0-9398-4cec-9b95-e510391864ab-kube-api-access-r98pf\") pod \"e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:08.379050 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.378967 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-eae138-openshift-default-dockercfg-kswgw\"" Nov 03 21:17:08.429454 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.429420 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7"] Nov 03 21:17:08.474572 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.474527 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f82eafd0-9398-4cec-9b95-e510391864ab-istio-podinfo\") pod \"e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:08.474866 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.474839 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-workload-certs\") pod \"e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:08.475062 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.475034 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-credential-socket\") pod \"e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:08.475318 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.475268 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-workload-socket\") pod \"e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:08.475524 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.475504 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-istio-envoy\") pod \"e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:08.477620 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.476708 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-workload-socket\") pod \"e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:08.478336 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.478310 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-workload-certs\") pod \"e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:08.478726 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.478662 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-credential-socket\") pod \"e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:08.478844 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.477554 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r98pf\" (UniqueName: \"kubernetes.io/projected/f82eafd0-9398-4cec-9b95-e510391864ab-kube-api-access-r98pf\") pod \"e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:08.479052 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.479034 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-istio-data\") pod \"e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:08.482685 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.479766 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f82eafd0-9398-4cec-9b95-e510391864ab-istio-token\") pod \"e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:08.483187 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.483159 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f82eafd0-9398-4cec-9b95-e510391864ab-istiod-ca-cert\") pod \"e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:08.485184 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.484020 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f82eafd0-9398-4cec-9b95-e510391864ab-istiod-ca-cert\") pod \"e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:08.485342 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.479710 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-istio-data\") pod \"e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:08.488577 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.488549 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f82eafd0-9398-4cec-9b95-e510391864ab-istio-podinfo\") pod \"e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:08.535189 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.535101 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-istio-envoy\") pod \"e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:08.565994 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.565920 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f82eafd0-9398-4cec-9b95-e510391864ab-istio-token\") pod \"e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:08.582678 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.582515 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r98pf\" (UniqueName: \"kubernetes.io/projected/f82eafd0-9398-4cec-9b95-e510391864ab-kube-api-access-r98pf\") pod \"e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:08.650413 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:08.650378 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:09.075923 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:09.075890 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7"] Nov 03 21:17:09.077060 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:17:09.077030 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf82eafd0_9398_4cec_9b95_e510391864ab.slice/crio-3ff87e4abe85f6040d609e7e1fd30ec03b320d248af20e0346342a70c5756a39 WatchSource:0}: Error finding container 3ff87e4abe85f6040d609e7e1fd30ec03b320d248af20e0346342a70c5756a39: Status 404 returned error can't find the container with id 3ff87e4abe85f6040d609e7e1fd30ec03b320d248af20e0346342a70c5756a39 Nov 03 21:17:09.079795 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:09.079751 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:17:09.079887 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:09.079850 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:17:09.079931 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:09.079897 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:17:09.401257 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:09.401169 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" event={"ID":"f82eafd0-9398-4cec-9b95-e510391864ab","Type":"ContainerStarted","Data":"efc7d29bc549dd372ddcb388afb5e8cadc82bf8e97142c1f562db32ff49fe629"} Nov 03 21:17:09.401257 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:09.401231 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" event={"ID":"f82eafd0-9398-4cec-9b95-e510391864ab","Type":"ContainerStarted","Data":"3ff87e4abe85f6040d609e7e1fd30ec03b320d248af20e0346342a70c5756a39"} Nov 03 21:17:09.537247 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:09.537169 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" podStartSLOduration=1.537149477 podStartE2EDuration="1.537149477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:17:09.533573136 +0000 UTC m=+1724.001351862" watchObservedRunningTime="2026-11-03 21:17:09.537149477 +0000 UTC m=+1724.004928202" Nov 03 21:17:09.682152 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:09.657783 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:10.619063 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:17:10.619020 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:17:10.656529 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:10.656497 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:11.002539 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:11.002475 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-gtbjq" podUID="cf675e88-d437-4eba-8bb5-d602bee8133f" containerName="payload-pre-processing" containerID="cri-o://390feca44a4873bb90afd01116e9d8bb6285dfe351bd866b80ad60e39c4df55a" gracePeriod=25 Nov 03 21:17:11.290362 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:11.290335 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-gtbjq" Nov 03 21:17:11.350492 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:11.350453 5615 patch_prober.go:29] interesting pod/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 21:17:11.350668 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:11.350508 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" podUID="1310ac80-c9fa-4498-8bad-6be0ded4c495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 21:17:11.366898 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:11.366855 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf675e88-d437-4eba-8bb5-d602bee8133f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf675e88-d437-4eba-8bb5-d602bee8133f-plugins-config-volume\") pod \"cf675e88-d437-4eba-8bb5-d602bee8133f\" (UID: \"cf675e88-d437-4eba-8bb5-d602bee8133f\") " Nov 03 21:17:11.367262 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:11.367243 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf675e88-d437-4eba-8bb5-d602bee8133f-kube-api-access-t24vf\" (UniqueName: \"kubernetes.io/projected/cf675e88-d437-4eba-8bb5-d602bee8133f-kube-api-access-t24vf\") pod \"cf675e88-d437-4eba-8bb5-d602bee8133f\" (UID: \"cf675e88-d437-4eba-8bb5-d602bee8133f\") " Nov 03 21:17:11.367349 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:11.367272 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf675e88-d437-4eba-8bb5-d602bee8133f-plugins-config-volume" pod "cf675e88-d437-4eba-8bb5-d602bee8133f" (UID: "cf675e88-d437-4eba-8bb5-d602bee8133f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:17:11.367681 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:11.367659 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf675e88-d437-4eba-8bb5-d602bee8133f-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:11.369397 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:11.369374 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf675e88-d437-4eba-8bb5-d602bee8133f-kube-api-access-t24vf" pod "cf675e88-d437-4eba-8bb5-d602bee8133f" (UID: "cf675e88-d437-4eba-8bb5-d602bee8133f"). InnerVolumeSpecName "kube-api-access-t24vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:11.419210 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:11.419177 5615 generic.go:356] "Generic (PLEG): container finished" podID="1310ac80-c9fa-4498-8bad-6be0ded4c495" containerID="dc18fcaae9950171f61627ff17264f9ac5dd80f3e63812ec6c525f146a3b9a08" exitCode=0 Nov 03 21:17:11.419393 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:11.419257 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" event={"ID":"1310ac80-c9fa-4498-8bad-6be0ded4c495","Type":"ContainerDied","Data":"dc18fcaae9950171f61627ff17264f9ac5dd80f3e63812ec6c525f146a3b9a08"} Nov 03 21:17:11.420195 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:11.419800 5615 scope.go:122] "RemoveContainer" containerID="dc18fcaae9950171f61627ff17264f9ac5dd80f3e63812ec6c525f146a3b9a08" Nov 03 21:17:11.421088 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:11.421044 5615 generic.go:356] "Generic (PLEG): container finished" podID="cf675e88-d437-4eba-8bb5-d602bee8133f" containerID="390feca44a4873bb90afd01116e9d8bb6285dfe351bd866b80ad60e39c4df55a" exitCode=0 Nov 03 21:17:11.421215 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:11.421107 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-gtbjq" Nov 03 21:17:11.421215 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:11.421137 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-gtbjq" event={"ID":"cf675e88-d437-4eba-8bb5-d602bee8133f","Type":"ContainerDied","Data":"390feca44a4873bb90afd01116e9d8bb6285dfe351bd866b80ad60e39c4df55a"} Nov 03 21:17:11.421334 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:11.421215 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-gtbjq" event={"ID":"cf675e88-d437-4eba-8bb5-d602bee8133f","Type":"ContainerDied","Data":"7671c450505b6fb667a1dab822457d9c10a400972ae5c8598aa3c83b04622512"} Nov 03 21:17:11.421334 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:11.421248 5615 scope.go:122] "RemoveContainer" containerID="390feca44a4873bb90afd01116e9d8bb6285dfe351bd866b80ad60e39c4df55a" Nov 03 21:17:11.421810 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:11.421675 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:11.421810 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:11.421685 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:17:11.421810 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:11.421770 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:17:11.421984 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:11.421822 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:17:11.423083 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:11.422981 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:17:11.444121 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:11.444059 5615 scope.go:122] "RemoveContainer" containerID="390feca44a4873bb90afd01116e9d8bb6285dfe351bd866b80ad60e39c4df55a" Nov 03 21:17:11.444595 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:17:11.444568 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"390feca44a4873bb90afd01116e9d8bb6285dfe351bd866b80ad60e39c4df55a\": container with ID starting with 390feca44a4873bb90afd01116e9d8bb6285dfe351bd866b80ad60e39c4df55a not found: ID does not exist" containerID="390feca44a4873bb90afd01116e9d8bb6285dfe351bd866b80ad60e39c4df55a" Nov 03 21:17:11.444696 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:11.444605 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"390feca44a4873bb90afd01116e9d8bb6285dfe351bd866b80ad60e39c4df55a"} err="failed to get container status \"390feca44a4873bb90afd01116e9d8bb6285dfe351bd866b80ad60e39c4df55a\": rpc error: code = NotFound desc = could not find container \"390feca44a4873bb90afd01116e9d8bb6285dfe351bd866b80ad60e39c4df55a\": container with ID starting with 390feca44a4873bb90afd01116e9d8bb6285dfe351bd866b80ad60e39c4df55a not found: ID does not exist" Nov 03 21:17:11.469830 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:11.469787 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t24vf\" (UniqueName: \"kubernetes.io/projected/cf675e88-d437-4eba-8bb5-d602bee8133f-kube-api-access-t24vf\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:11.761086 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:11.761041 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-gtbjq"] Nov 03 21:17:11.782205 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:11.782164 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-gtbjq"] Nov 03 21:17:12.432584 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:12.432544 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" event={"ID":"1310ac80-c9fa-4498-8bad-6be0ded4c495","Type":"ContainerStarted","Data":"62ee231e6df2e92b8acfa11f39aa326fe018a16ad0013d51e9a2752b34f2ae36"} Nov 03 21:17:12.654179 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:12.654144 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:17:12.654377 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:12.654357 5615 patch_prober.go:29] interesting pod/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 21:17:12.654423 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:12.654400 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" podUID="1310ac80-c9fa-4498-8bad-6be0ded4c495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 21:17:13.344488 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:13.344460 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-79d96df5c6-cfzcv" Nov 03 21:17:13.653389 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:13.653313 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf675e88-d437-4eba-8bb5-d602bee8133f" path="/var/lib/kubelet/pods/cf675e88-d437-4eba-8bb5-d602bee8133f/volumes" Nov 03 21:17:13.653877 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:13.653855 5615 patch_prober.go:29] interesting pod/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 21:17:13.653928 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:13.653901 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" podUID="1310ac80-c9fa-4498-8bad-6be0ded4c495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 21:17:14.234635 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:14.234592 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-7468b6b967-4nh5h"] Nov 03 21:17:14.351544 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:14.351514 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-b7cdc65f9-vsbx8" Nov 03 21:17:14.683801 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:14.683713 5615 patch_prober.go:29] interesting pod/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 21:17:14.684106 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:14.684050 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" podUID="1310ac80-c9fa-4498-8bad-6be0ded4c495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 21:17:15.664679 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:15.662275 5615 patch_prober.go:29] interesting pod/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 21:17:15.664679 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:15.662352 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" podUID="1310ac80-c9fa-4498-8bad-6be0ded4c495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 21:17:16.653658 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:16.653618 5615 patch_prober.go:29] interesting pod/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 21:17:16.653871 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:16.653683 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" podUID="1310ac80-c9fa-4498-8bad-6be0ded4c495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 21:17:17.750668 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:17.750634 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:17:17.751646 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:17.751618 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:17:17.752540 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:17.752516 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:17:19.235271 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:19.235227 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-7468b6b967-4nh5h" podUID="79c93b60-376b-4176-86b9-e786ec8334b4" containerName="payload-pre-processing" containerID="cri-o://ea8dee4e3d0da701a8b2aeb41f2a6b9448f8341559353f7732a807259a0646d4" gracePeriod=25 Nov 03 21:17:19.506448 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:19.506422 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-7468b6b967-4nh5h" Nov 03 21:17:19.541714 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:19.541402 5615 generic.go:356] "Generic (PLEG): container finished" podID="79c93b60-376b-4176-86b9-e786ec8334b4" containerID="ea8dee4e3d0da701a8b2aeb41f2a6b9448f8341559353f7732a807259a0646d4" exitCode=0 Nov 03 21:17:19.541714 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:19.541469 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-7468b6b967-4nh5h" Nov 03 21:17:19.541714 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:19.541531 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-7468b6b967-4nh5h" event={"ID":"79c93b60-376b-4176-86b9-e786ec8334b4","Type":"ContainerDied","Data":"ea8dee4e3d0da701a8b2aeb41f2a6b9448f8341559353f7732a807259a0646d4"} Nov 03 21:17:19.541714 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:19.541580 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-7468b6b967-4nh5h" event={"ID":"79c93b60-376b-4176-86b9-e786ec8334b4","Type":"ContainerDied","Data":"e681158fef749733b4f55e4be1079a9738f1b57a528ae490a789c0095cc3a932"} Nov 03 21:17:19.541714 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:19.541602 5615 scope.go:122] "RemoveContainer" containerID="ea8dee4e3d0da701a8b2aeb41f2a6b9448f8341559353f7732a807259a0646d4" Nov 03 21:17:19.553337 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:19.553315 5615 scope.go:122] "RemoveContainer" containerID="ea8dee4e3d0da701a8b2aeb41f2a6b9448f8341559353f7732a807259a0646d4" Nov 03 21:17:19.553701 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:17:19.553676 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea8dee4e3d0da701a8b2aeb41f2a6b9448f8341559353f7732a807259a0646d4\": container with ID starting with ea8dee4e3d0da701a8b2aeb41f2a6b9448f8341559353f7732a807259a0646d4 not found: ID does not exist" containerID="ea8dee4e3d0da701a8b2aeb41f2a6b9448f8341559353f7732a807259a0646d4" Nov 03 21:17:19.553795 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:19.553708 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea8dee4e3d0da701a8b2aeb41f2a6b9448f8341559353f7732a807259a0646d4"} err="failed to get container status \"ea8dee4e3d0da701a8b2aeb41f2a6b9448f8341559353f7732a807259a0646d4\": rpc error: code = NotFound desc = could not find container \"ea8dee4e3d0da701a8b2aeb41f2a6b9448f8341559353f7732a807259a0646d4\": container with ID starting with ea8dee4e3d0da701a8b2aeb41f2a6b9448f8341559353f7732a807259a0646d4 not found: ID does not exist" Nov 03 21:17:19.572569 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:19.572535 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79c93b60-376b-4176-86b9-e786ec8334b4-kube-api-access-hzl8n\" (UniqueName: \"kubernetes.io/projected/79c93b60-376b-4176-86b9-e786ec8334b4-kube-api-access-hzl8n\") pod \"79c93b60-376b-4176-86b9-e786ec8334b4\" (UID: \"79c93b60-376b-4176-86b9-e786ec8334b4\") " Nov 03 21:17:19.572749 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:19.572605 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79c93b60-376b-4176-86b9-e786ec8334b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79c93b60-376b-4176-86b9-e786ec8334b4-plugins-config-volume\") pod \"79c93b60-376b-4176-86b9-e786ec8334b4\" (UID: \"79c93b60-376b-4176-86b9-e786ec8334b4\") " Nov 03 21:17:19.573093 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:19.573051 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c93b60-376b-4176-86b9-e786ec8334b4-plugins-config-volume" pod "79c93b60-376b-4176-86b9-e786ec8334b4" (UID: "79c93b60-376b-4176-86b9-e786ec8334b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:17:19.573495 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:19.573450 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79c93b60-376b-4176-86b9-e786ec8334b4-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:19.575429 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:19.575405 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c93b60-376b-4176-86b9-e786ec8334b4-kube-api-access-hzl8n" pod "79c93b60-376b-4176-86b9-e786ec8334b4" (UID: "79c93b60-376b-4176-86b9-e786ec8334b4"). InnerVolumeSpecName "kube-api-access-hzl8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:19.674560 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:19.674530 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzl8n\" (UniqueName: \"kubernetes.io/projected/79c93b60-376b-4176-86b9-e786ec8334b4-kube-api-access-hzl8n\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:19.951450 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:19.951365 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-7468b6b967-4nh5h"] Nov 03 21:17:19.960039 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:19.960007 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-7468b6b967-4nh5h"] Nov 03 21:17:21.653104 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:21.653053 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79c93b60-376b-4176-86b9-e786ec8334b4" path="/var/lib/kubelet/pods/79c93b60-376b-4176-86b9-e786ec8334b4/volumes" Nov 03 21:17:33.655155 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:33.655117 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc"] Nov 03 21:17:33.655718 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:33.655349 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" podUID="12f336dd-0843-4263-90b8-25db7598d76f" containerName="istio-proxy" containerID="cri-o://75abec1ae39fa222452f5820e7329ad71aa06457614847ff54c40ef13045c5cd" gracePeriod=30 Nov 03 21:17:38.703827 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:38.703798 5615 generic.go:356] "Generic (PLEG): container finished" podID="12f336dd-0843-4263-90b8-25db7598d76f" containerID="75abec1ae39fa222452f5820e7329ad71aa06457614847ff54c40ef13045c5cd" exitCode=0 Nov 03 21:17:38.704640 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:38.704425 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" event={"ID":"12f336dd-0843-4263-90b8-25db7598d76f","Type":"ContainerDied","Data":"75abec1ae39fa222452f5820e7329ad71aa06457614847ff54c40ef13045c5cd"} Nov 03 21:17:38.879252 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:38.879222 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:17:38.993438 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:38.993407 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-istio-data\") pod \"12f336dd-0843-4263-90b8-25db7598d76f\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " Nov 03 21:17:38.993644 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:38.993465 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12f336dd-0843-4263-90b8-25db7598d76f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f336dd-0843-4263-90b8-25db7598d76f-istiod-ca-cert\") pod \"12f336dd-0843-4263-90b8-25db7598d76f\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " Nov 03 21:17:38.993644 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:38.993525 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-credential-socket\") pod \"12f336dd-0843-4263-90b8-25db7598d76f\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " Nov 03 21:17:38.993644 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:38.993576 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12f336dd-0843-4263-90b8-25db7598d76f-istio-token\" (UniqueName: \"kubernetes.io/projected/12f336dd-0843-4263-90b8-25db7598d76f-istio-token\") pod \"12f336dd-0843-4263-90b8-25db7598d76f\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " Nov 03 21:17:38.993817 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:38.993645 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-workload-certs\") pod \"12f336dd-0843-4263-90b8-25db7598d76f\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " Nov 03 21:17:38.993817 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:38.993682 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-istio-envoy\") pod \"12f336dd-0843-4263-90b8-25db7598d76f\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " Nov 03 21:17:38.993817 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:38.993725 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-istio-data" pod "12f336dd-0843-4263-90b8-25db7598d76f" (UID: "12f336dd-0843-4263-90b8-25db7598d76f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:38.993968 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:38.993852 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-credential-socket" pod "12f336dd-0843-4263-90b8-25db7598d76f" (UID: "12f336dd-0843-4263-90b8-25db7598d76f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:38.993968 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:38.993872 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-workload-socket\") pod \"12f336dd-0843-4263-90b8-25db7598d76f\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " Nov 03 21:17:38.993968 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:38.993937 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-workload-certs" pod "12f336dd-0843-4263-90b8-25db7598d76f" (UID: "12f336dd-0843-4263-90b8-25db7598d76f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:38.994140 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:38.993986 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12f336dd-0843-4263-90b8-25db7598d76f-istiod-ca-cert" pod "12f336dd-0843-4263-90b8-25db7598d76f" (UID: "12f336dd-0843-4263-90b8-25db7598d76f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:17:38.994194 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:38.994048 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-workload-socket" pod "12f336dd-0843-4263-90b8-25db7598d76f" (UID: "12f336dd-0843-4263-90b8-25db7598d76f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:38.994194 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:38.993996 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12f336dd-0843-4263-90b8-25db7598d76f-kube-api-access-vl6g7\" (UniqueName: \"kubernetes.io/projected/12f336dd-0843-4263-90b8-25db7598d76f-kube-api-access-vl6g7\") pod \"12f336dd-0843-4263-90b8-25db7598d76f\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " Nov 03 21:17:38.994301 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:38.994230 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12f336dd-0843-4263-90b8-25db7598d76f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f336dd-0843-4263-90b8-25db7598d76f-istio-podinfo\") pod \"12f336dd-0843-4263-90b8-25db7598d76f\" (UID: \"12f336dd-0843-4263-90b8-25db7598d76f\") " Nov 03 21:17:38.994772 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:38.994750 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:38.994856 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:38.994776 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:38.994856 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:38.994789 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:38.994856 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:38.994801 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:38.994856 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:38.994814 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f336dd-0843-4263-90b8-25db7598d76f-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:38.995999 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:38.995970 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-istio-envoy" pod "12f336dd-0843-4263-90b8-25db7598d76f" (UID: "12f336dd-0843-4263-90b8-25db7598d76f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:38.996143 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:38.996133 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12f336dd-0843-4263-90b8-25db7598d76f-istio-podinfo" pod "12f336dd-0843-4263-90b8-25db7598d76f" (UID: "12f336dd-0843-4263-90b8-25db7598d76f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:17:38.996389 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:38.996362 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f336dd-0843-4263-90b8-25db7598d76f-istio-token" pod "12f336dd-0843-4263-90b8-25db7598d76f" (UID: "12f336dd-0843-4263-90b8-25db7598d76f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:38.996447 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:38.996402 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f336dd-0843-4263-90b8-25db7598d76f-kube-api-access-vl6g7" pod "12f336dd-0843-4263-90b8-25db7598d76f" (UID: "12f336dd-0843-4263-90b8-25db7598d76f"). InnerVolumeSpecName "kube-api-access-vl6g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:39.096470 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:39.096433 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl6g7\" (UniqueName: \"kubernetes.io/projected/12f336dd-0843-4263-90b8-25db7598d76f-kube-api-access-vl6g7\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:39.096470 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:39.096467 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f336dd-0843-4263-90b8-25db7598d76f-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:39.096470 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:39.096477 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12f336dd-0843-4263-90b8-25db7598d76f-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:39.096706 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:39.096488 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f336dd-0843-4263-90b8-25db7598d76f-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:39.712462 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:39.712424 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" Nov 03 21:17:39.712871 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:39.712463 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc" event={"ID":"12f336dd-0843-4263-90b8-25db7598d76f","Type":"ContainerDied","Data":"604190aa0f5693f46cb3fc9b719130c4191afc2bfbeae8e23735fdb1d0ad7472"} Nov 03 21:17:39.712871 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:39.712524 5615 scope.go:122] "RemoveContainer" containerID="75abec1ae39fa222452f5820e7329ad71aa06457614847ff54c40ef13045c5cd" Nov 03 21:17:39.917921 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:39.917833 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc"] Nov 03 21:17:39.964940 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:39.964901 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-825tc"] Nov 03 21:17:40.618911 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:17:40.618875 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:17:41.654320 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:41.654282 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12f336dd-0843-4263-90b8-25db7598d76f" path="/var/lib/kubelet/pods/12f336dd-0843-4263-90b8-25db7598d76f/volumes" Nov 03 21:17:44.637157 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:44.637120 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-75b98768b4-v4jxs"] Nov 03 21:17:44.637821 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:44.637806 5615 cpu_manager.go:414] "containerMap: removing container" podUID="79c93b60-376b-4176-86b9-e786ec8334b4" containerName="payload-pre-processing" Nov 03 21:17:44.637874 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:44.637822 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79c93b60-376b-4176-86b9-e786ec8334b4" containerName="payload-pre-processing" Nov 03 21:17:44.637874 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:44.637836 5615 cpu_manager.go:414] "containerMap: removing container" podUID="12f336dd-0843-4263-90b8-25db7598d76f" containerName="istio-proxy" Nov 03 21:17:44.637874 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:44.637841 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12f336dd-0843-4263-90b8-25db7598d76f" containerName="istio-proxy" Nov 03 21:17:44.637874 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:44.637851 5615 cpu_manager.go:414] "containerMap: removing container" podUID="cf675e88-d437-4eba-8bb5-d602bee8133f" containerName="payload-pre-processing" Nov 03 21:17:44.637874 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:44.637856 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf675e88-d437-4eba-8bb5-d602bee8133f" containerName="payload-pre-processing" Nov 03 21:17:44.638028 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:44.637966 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="79c93b60-376b-4176-86b9-e786ec8334b4" containerName="payload-pre-processing" Nov 03 21:17:44.638028 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:44.637973 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="12f336dd-0843-4263-90b8-25db7598d76f" containerName="istio-proxy" Nov 03 21:17:44.638028 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:44.637979 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf675e88-d437-4eba-8bb5-d602bee8133f" containerName="payload-pre-processing" Nov 03 21:17:44.642227 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:44.642203 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-75b98768b4-v4jxs" Nov 03 21:17:44.692442 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:44.692394 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-812243\"" Nov 03 21:17:44.711573 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:44.711523 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8t8r\" (UniqueName: \"kubernetes.io/projected/e52dbc1f-a800-4c31-b7ec-01210231d982-kube-api-access-s8t8r\") pod \"payload-pre-processing-e2e-ipp-a-812243-75b98768b4-v4jxs\" (UID: \"e52dbc1f-a800-4c31-b7ec-01210231d982\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-75b98768b4-v4jxs" Nov 03 21:17:44.711781 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:44.711744 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52dbc1f-a800-4c31-b7ec-01210231d982-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-812243-75b98768b4-v4jxs\" (UID: \"e52dbc1f-a800-4c31-b7ec-01210231d982\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-75b98768b4-v4jxs" Nov 03 21:17:44.739329 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:44.739301 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-812243-dockercfg-sxcw7\"" Nov 03 21:17:44.742430 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:44.742402 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-75b98768b4-v4jxs"] Nov 03 21:17:44.813524 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:44.813487 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52dbc1f-a800-4c31-b7ec-01210231d982-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-812243-75b98768b4-v4jxs\" (UID: \"e52dbc1f-a800-4c31-b7ec-01210231d982\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-75b98768b4-v4jxs" Nov 03 21:17:44.813748 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:44.813641 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8t8r\" (UniqueName: \"kubernetes.io/projected/e52dbc1f-a800-4c31-b7ec-01210231d982-kube-api-access-s8t8r\") pod \"payload-pre-processing-e2e-ipp-a-812243-75b98768b4-v4jxs\" (UID: \"e52dbc1f-a800-4c31-b7ec-01210231d982\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-75b98768b4-v4jxs" Nov 03 21:17:44.814292 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:44.814265 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52dbc1f-a800-4c31-b7ec-01210231d982-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-812243-75b98768b4-v4jxs\" (UID: \"e52dbc1f-a800-4c31-b7ec-01210231d982\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-75b98768b4-v4jxs" Nov 03 21:17:44.872600 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:44.872567 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8t8r\" (UniqueName: \"kubernetes.io/projected/e52dbc1f-a800-4c31-b7ec-01210231d982-kube-api-access-s8t8r\") pod \"payload-pre-processing-e2e-ipp-a-812243-75b98768b4-v4jxs\" (UID: \"e52dbc1f-a800-4c31-b7ec-01210231d982\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-75b98768b4-v4jxs" Nov 03 21:17:44.955759 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:44.955669 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-75b98768b4-v4jxs" Nov 03 21:17:45.328609 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:45.328579 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-75b98768b4-v4jxs"] Nov 03 21:17:45.330737 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:17:45.330700 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode52dbc1f_a800_4c31_b7ec_01210231d982.slice/crio-5af7e75aaa47ee851e2546627622c82ade59a64e5b482bb374ef0de1ed50f8e9 WatchSource:0}: Error finding container 5af7e75aaa47ee851e2546627622c82ade59a64e5b482bb374ef0de1ed50f8e9: Status 404 returned error can't find the container with id 5af7e75aaa47ee851e2546627622c82ade59a64e5b482bb374ef0de1ed50f8e9 Nov 03 21:17:45.454272 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:45.454239 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-812243-67b64449fd-ddqlz"] Nov 03 21:17:45.458033 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:45.458005 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-67b64449fd-ddqlz" Nov 03 21:17:45.520796 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:45.520750 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-812243-67b64449fd-ddqlz"] Nov 03 21:17:45.633967 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:45.633881 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ceab91d-c4f1-4eb0-baf0-753f0ac73ede-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-812243-67b64449fd-ddqlz\" (UID: \"8ceab91d-c4f1-4eb0-baf0-753f0ac73ede\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-67b64449fd-ddqlz" Nov 03 21:17:45.633967 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:45.633937 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d596g\" (UniqueName: \"kubernetes.io/projected/8ceab91d-c4f1-4eb0-baf0-753f0ac73ede-kube-api-access-d596g\") pod \"payload-processing-e2e-ipp-a-812243-67b64449fd-ddqlz\" (UID: \"8ceab91d-c4f1-4eb0-baf0-753f0ac73ede\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-67b64449fd-ddqlz" Nov 03 21:17:45.735360 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:45.735321 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ceab91d-c4f1-4eb0-baf0-753f0ac73ede-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-812243-67b64449fd-ddqlz\" (UID: \"8ceab91d-c4f1-4eb0-baf0-753f0ac73ede\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-67b64449fd-ddqlz" Nov 03 21:17:45.735744 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:45.735379 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d596g\" (UniqueName: \"kubernetes.io/projected/8ceab91d-c4f1-4eb0-baf0-753f0ac73ede-kube-api-access-d596g\") pod \"payload-processing-e2e-ipp-a-812243-67b64449fd-ddqlz\" (UID: \"8ceab91d-c4f1-4eb0-baf0-753f0ac73ede\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-67b64449fd-ddqlz" Nov 03 21:17:45.736064 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:45.736042 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ceab91d-c4f1-4eb0-baf0-753f0ac73ede-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-812243-67b64449fd-ddqlz\" (UID: \"8ceab91d-c4f1-4eb0-baf0-753f0ac73ede\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-67b64449fd-ddqlz" Nov 03 21:17:45.756125 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:45.756089 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-75b98768b4-v4jxs" event={"ID":"e52dbc1f-a800-4c31-b7ec-01210231d982","Type":"ContainerStarted","Data":"0be767a38848a16c208ae24042ccbb637c1ed346002cc28904c8659df16c0ad5"} Nov 03 21:17:45.756304 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:45.756137 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-75b98768b4-v4jxs" event={"ID":"e52dbc1f-a800-4c31-b7ec-01210231d982","Type":"ContainerStarted","Data":"5af7e75aaa47ee851e2546627622c82ade59a64e5b482bb374ef0de1ed50f8e9"} Nov 03 21:17:45.756304 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:45.756192 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-75b98768b4-v4jxs" Nov 03 21:17:45.793738 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:45.793702 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d596g\" (UniqueName: \"kubernetes.io/projected/8ceab91d-c4f1-4eb0-baf0-753f0ac73ede-kube-api-access-d596g\") pod \"payload-processing-e2e-ipp-a-812243-67b64449fd-ddqlz\" (UID: \"8ceab91d-c4f1-4eb0-baf0-753f0ac73ede\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-67b64449fd-ddqlz" Nov 03 21:17:46.071130 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:46.071097 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-67b64449fd-ddqlz" Nov 03 21:17:46.448748 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:46.448689 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-75b98768b4-v4jxs" podStartSLOduration=2.448670632 podStartE2EDuration="2.448670632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:17:45.867238101 +0000 UTC m=+1760.335016825" watchObservedRunningTime="2026-11-03 21:17:46.448670632 +0000 UTC m=+1760.916449361" Nov 03 21:17:46.451283 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:46.451244 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-812243-67b64449fd-ddqlz"] Nov 03 21:17:46.452093 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:17:46.452049 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ceab91d_c4f1_4eb0_baf0_753f0ac73ede.slice/crio-2aa5a93c0d251b934b3f9d310b6e58f4b765d2922fc8e613505ec126010284fa WatchSource:0}: Error finding container 2aa5a93c0d251b934b3f9d310b6e58f4b765d2922fc8e613505ec126010284fa: Status 404 returned error can't find the container with id 2aa5a93c0d251b934b3f9d310b6e58f4b765d2922fc8e613505ec126010284fa Nov 03 21:17:46.765241 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:46.765204 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-67b64449fd-ddqlz" event={"ID":"8ceab91d-c4f1-4eb0-baf0-753f0ac73ede","Type":"ContainerStarted","Data":"14cfa4c7f8f585aac3760d9bb6dde50836839e0904b6a77eb7d05dd70d81da08"} Nov 03 21:17:46.765704 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:46.765260 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-67b64449fd-ddqlz" event={"ID":"8ceab91d-c4f1-4eb0-baf0-753f0ac73ede","Type":"ContainerStarted","Data":"2aa5a93c0d251b934b3f9d310b6e58f4b765d2922fc8e613505ec126010284fa"} Nov 03 21:17:46.765704 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:46.765468 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-67b64449fd-ddqlz" Nov 03 21:17:46.773275 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:46.773245 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9232e5-7b98c54cc4-ngvtb"] Nov 03 21:17:46.879719 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:46.879659 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-67b64449fd-ddqlz" podStartSLOduration=1.879645027 podStartE2EDuration="1.879645027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:17:46.877303815 +0000 UTC m=+1761.345082539" watchObservedRunningTime="2026-11-03 21:17:46.879645027 +0000 UTC m=+1761.347423749" Nov 03 21:17:51.773922 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:51.773876 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-9232e5-7b98c54cc4-ngvtb" podUID="a7906cbf-9a94-4fe8-952d-48fed7a1cd3e" containerName="payload-processing" containerID="cri-o://c71e46f314e497f17dec4ac0d4454b6081de0ab7246b2c64bf4b0c56658b5d2b" gracePeriod=25 Nov 03 21:17:52.043590 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:52.043518 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9232e5-7b98c54cc4-ngvtb" Nov 03 21:17:52.164631 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:52.164595 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7906cbf-9a94-4fe8-952d-48fed7a1cd3e-kube-api-access-mm7rn\" (UniqueName: \"kubernetes.io/projected/a7906cbf-9a94-4fe8-952d-48fed7a1cd3e-kube-api-access-mm7rn\") pod \"a7906cbf-9a94-4fe8-952d-48fed7a1cd3e\" (UID: \"a7906cbf-9a94-4fe8-952d-48fed7a1cd3e\") " Nov 03 21:17:52.164917 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:52.164900 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7906cbf-9a94-4fe8-952d-48fed7a1cd3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7906cbf-9a94-4fe8-952d-48fed7a1cd3e-plugins-config-volume\") pod \"a7906cbf-9a94-4fe8-952d-48fed7a1cd3e\" (UID: \"a7906cbf-9a94-4fe8-952d-48fed7a1cd3e\") " Nov 03 21:17:52.165271 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:52.165245 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7906cbf-9a94-4fe8-952d-48fed7a1cd3e-plugins-config-volume" pod "a7906cbf-9a94-4fe8-952d-48fed7a1cd3e" (UID: "a7906cbf-9a94-4fe8-952d-48fed7a1cd3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:17:52.165460 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:52.165445 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7906cbf-9a94-4fe8-952d-48fed7a1cd3e-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:52.167123 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:52.167091 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7906cbf-9a94-4fe8-952d-48fed7a1cd3e-kube-api-access-mm7rn" pod "a7906cbf-9a94-4fe8-952d-48fed7a1cd3e" (UID: "a7906cbf-9a94-4fe8-952d-48fed7a1cd3e"). InnerVolumeSpecName "kube-api-access-mm7rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:52.267452 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:52.267410 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm7rn\" (UniqueName: \"kubernetes.io/projected/a7906cbf-9a94-4fe8-952d-48fed7a1cd3e-kube-api-access-mm7rn\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:52.807169 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:52.807126 5615 generic.go:356] "Generic (PLEG): container finished" podID="a7906cbf-9a94-4fe8-952d-48fed7a1cd3e" containerID="c71e46f314e497f17dec4ac0d4454b6081de0ab7246b2c64bf4b0c56658b5d2b" exitCode=0 Nov 03 21:17:52.807644 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:52.807166 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9232e5-7b98c54cc4-ngvtb" event={"ID":"a7906cbf-9a94-4fe8-952d-48fed7a1cd3e","Type":"ContainerDied","Data":"c71e46f314e497f17dec4ac0d4454b6081de0ab7246b2c64bf4b0c56658b5d2b"} Nov 03 21:17:52.807644 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:52.807193 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9232e5-7b98c54cc4-ngvtb" Nov 03 21:17:52.807644 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:52.807226 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9232e5-7b98c54cc4-ngvtb" event={"ID":"a7906cbf-9a94-4fe8-952d-48fed7a1cd3e","Type":"ContainerDied","Data":"a104166eb7d6f3f0bcbdf30341fd284f099fa1da04120ebdfa0aa8dc0abf2c40"} Nov 03 21:17:52.807644 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:52.807255 5615 scope.go:122] "RemoveContainer" containerID="c71e46f314e497f17dec4ac0d4454b6081de0ab7246b2c64bf4b0c56658b5d2b" Nov 03 21:17:52.818436 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:52.818416 5615 scope.go:122] "RemoveContainer" containerID="c71e46f314e497f17dec4ac0d4454b6081de0ab7246b2c64bf4b0c56658b5d2b" Nov 03 21:17:52.818735 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:17:52.818717 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c71e46f314e497f17dec4ac0d4454b6081de0ab7246b2c64bf4b0c56658b5d2b\": container with ID starting with c71e46f314e497f17dec4ac0d4454b6081de0ab7246b2c64bf4b0c56658b5d2b not found: ID does not exist" containerID="c71e46f314e497f17dec4ac0d4454b6081de0ab7246b2c64bf4b0c56658b5d2b" Nov 03 21:17:52.818813 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:52.818740 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c71e46f314e497f17dec4ac0d4454b6081de0ab7246b2c64bf4b0c56658b5d2b"} err="failed to get container status \"c71e46f314e497f17dec4ac0d4454b6081de0ab7246b2c64bf4b0c56658b5d2b\": rpc error: code = NotFound desc = could not find container \"c71e46f314e497f17dec4ac0d4454b6081de0ab7246b2c64bf4b0c56658b5d2b\": container with ID starting with c71e46f314e497f17dec4ac0d4454b6081de0ab7246b2c64bf4b0c56658b5d2b not found: ID does not exist" Nov 03 21:17:52.966985 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:52.966948 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9232e5-7b98c54cc4-ngvtb"] Nov 03 21:17:53.013912 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:53.013878 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9232e5-7b98c54cc4-ngvtb"] Nov 03 21:17:53.653310 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:53.653278 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7906cbf-9a94-4fe8-952d-48fed7a1cd3e" path="/var/lib/kubelet/pods/a7906cbf-9a94-4fe8-952d-48fed7a1cd3e/volumes" Nov 03 21:17:56.768048 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:56.768017 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-75b98768b4-v4jxs" Nov 03 21:17:57.775197 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:17:57.775167 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-67b64449fd-ddqlz" Nov 03 21:18:05.513894 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:05.513855 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-66513c-7765fcd666-6l4wb"] Nov 03 21:18:05.514563 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:05.514548 5615 cpu_manager.go:414] "containerMap: removing container" podUID="a7906cbf-9a94-4fe8-952d-48fed7a1cd3e" containerName="payload-processing" Nov 03 21:18:05.514615 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:05.514564 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7906cbf-9a94-4fe8-952d-48fed7a1cd3e" containerName="payload-processing" Nov 03 21:18:05.514687 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:05.514677 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7906cbf-9a94-4fe8-952d-48fed7a1cd3e" containerName="payload-processing" Nov 03 21:18:05.518459 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:05.518440 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-7765fcd666-6l4wb" Nov 03 21:18:05.540569 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:05.540525 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-66513c-7765fcd666-6l4wb"] Nov 03 21:18:05.568304 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:05.568269 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-66513c\"" Nov 03 21:18:05.568494 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:05.568321 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-66513c-dockercfg-wd8bs\"" Nov 03 21:18:05.683903 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:05.683868 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5wr7\" (UniqueName: \"kubernetes.io/projected/911bcfcb-bd59-4477-8b70-02f725d43ff7-kube-api-access-q5wr7\") pod \"payload-processing-e2e-ipp-a-66513c-7765fcd666-6l4wb\" (UID: \"911bcfcb-bd59-4477-8b70-02f725d43ff7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-7765fcd666-6l4wb" Nov 03 21:18:05.684049 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:05.683989 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/911bcfcb-bd59-4477-8b70-02f725d43ff7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-66513c-7765fcd666-6l4wb\" (UID: \"911bcfcb-bd59-4477-8b70-02f725d43ff7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-7765fcd666-6l4wb" Nov 03 21:18:05.785139 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:05.785015 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5wr7\" (UniqueName: \"kubernetes.io/projected/911bcfcb-bd59-4477-8b70-02f725d43ff7-kube-api-access-q5wr7\") pod \"payload-processing-e2e-ipp-a-66513c-7765fcd666-6l4wb\" (UID: \"911bcfcb-bd59-4477-8b70-02f725d43ff7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-7765fcd666-6l4wb" Nov 03 21:18:05.785493 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:05.785173 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/911bcfcb-bd59-4477-8b70-02f725d43ff7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-66513c-7765fcd666-6l4wb\" (UID: \"911bcfcb-bd59-4477-8b70-02f725d43ff7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-7765fcd666-6l4wb" Nov 03 21:18:05.786163 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:05.786142 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/911bcfcb-bd59-4477-8b70-02f725d43ff7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-66513c-7765fcd666-6l4wb\" (UID: \"911bcfcb-bd59-4477-8b70-02f725d43ff7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-7765fcd666-6l4wb" Nov 03 21:18:05.845632 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:05.845598 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5wr7\" (UniqueName: \"kubernetes.io/projected/911bcfcb-bd59-4477-8b70-02f725d43ff7-kube-api-access-q5wr7\") pod \"payload-processing-e2e-ipp-a-66513c-7765fcd666-6l4wb\" (UID: \"911bcfcb-bd59-4477-8b70-02f725d43ff7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-7765fcd666-6l4wb" Nov 03 21:18:06.132003 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:06.131914 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-7765fcd666-6l4wb" Nov 03 21:18:06.505335 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:06.505293 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-66513c-7765fcd666-6l4wb"] Nov 03 21:18:06.505936 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:18:06.505900 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911bcfcb_bd59_4477_8b70_02f725d43ff7.slice/crio-ac5c76af7c14c3822c26d5d8a9f976d329d11e7335b92adbe983a59ca0a68080 WatchSource:0}: Error finding container ac5c76af7c14c3822c26d5d8a9f976d329d11e7335b92adbe983a59ca0a68080: Status 404 returned error can't find the container with id ac5c76af7c14c3822c26d5d8a9f976d329d11e7335b92adbe983a59ca0a68080 Nov 03 21:18:06.977137 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:06.977021 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-7765fcd666-6l4wb" event={"ID":"911bcfcb-bd59-4477-8b70-02f725d43ff7","Type":"ContainerStarted","Data":"4c28ce2c3750835cd5e64e3f699c56a8f63b8812369a2a319683c28949b8200e"} Nov 03 21:18:06.977575 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:06.977137 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-7765fcd666-6l4wb" Nov 03 21:18:06.977575 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:06.977162 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-7765fcd666-6l4wb" event={"ID":"911bcfcb-bd59-4477-8b70-02f725d43ff7","Type":"ContainerStarted","Data":"ac5c76af7c14c3822c26d5d8a9f976d329d11e7335b92adbe983a59ca0a68080"} Nov 03 21:18:07.089784 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:07.089729 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-7765fcd666-6l4wb" podStartSLOduration=2.08971354 podStartE2EDuration="2.08971354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:18:07.087760538 +0000 UTC m=+1781.555539263" watchObservedRunningTime="2026-11-03 21:18:07.08971354 +0000 UTC m=+1781.557492262" Nov 03 21:18:10.619865 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:18:10.619818 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:18:15.800370 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:15.800335 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm"] Nov 03 21:18:15.800868 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:15.800603 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm" podUID="c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867" containerName="maas-api" containerID="cri-o://5c6f5dcac0507028db1510fd254d52ff4022dc6bc988b077f5fa3419bbd90d31" gracePeriod=30 Nov 03 21:18:16.135961 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:16.135325 5615 generic.go:356] "Generic (PLEG): container finished" podID="c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867" containerID="5c6f5dcac0507028db1510fd254d52ff4022dc6bc988b077f5fa3419bbd90d31" exitCode=0 Nov 03 21:18:16.135961 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:16.135595 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm" event={"ID":"c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867","Type":"ContainerDied","Data":"5c6f5dcac0507028db1510fd254d52ff4022dc6bc988b077f5fa3419bbd90d31"} Nov 03 21:18:16.157621 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:16.157266 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm" Nov 03 21:18:16.250464 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:16.250428 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-maas-api-tls\") pod \"c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867\" (UID: \"c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867\") " Nov 03 21:18:16.250646 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:16.250519 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-metrics-certs\") pod \"c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867\" (UID: \"c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867\") " Nov 03 21:18:16.250693 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:16.250675 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-kube-api-access-xw9qh\" (UniqueName: \"kubernetes.io/projected/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-kube-api-access-xw9qh\") pod \"c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867\" (UID: \"c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867\") " Nov 03 21:18:16.250738 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:16.250726 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-openshift-service-ca\") pod \"c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867\" (UID: \"c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867\") " Nov 03 21:18:16.251196 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:16.251170 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-openshift-service-ca" pod "c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867" (UID: "c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:18:16.252802 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:16.252778 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-kube-api-access-xw9qh" pod "c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867" (UID: "c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867"). InnerVolumeSpecName "kube-api-access-xw9qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:18:16.252902 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:16.252842 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-metrics-certs" pod "c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867" (UID: "c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:18:16.252902 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:16.252851 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-maas-api-tls" pod "c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867" (UID: "c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:18:16.290665 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:16.290620 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-78b68f9566-6hnx6"] Nov 03 21:18:16.290860 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:16.290826 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-78b68f9566-6hnx6" podUID="95aee87a-8056-47a5-9574-02d25a20f331" containerName="payload-pre-processing" containerID="cri-o://02e2a94d8703d085d7951fd65c8539de2e9a80a69b928c32c347318d48b355fc" gracePeriod=30 Nov 03 21:18:16.352570 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:16.352536 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw9qh\" (UniqueName: \"kubernetes.io/projected/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-kube-api-access-xw9qh\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:16.352570 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:16.352568 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:16.352770 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:16.352581 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:16.352770 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:16.352591 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:16.549017 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:16.548990 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-78b68f9566-6hnx6" Nov 03 21:18:16.656115 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:16.656061 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95aee87a-8056-47a5-9574-02d25a20f331-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95aee87a-8056-47a5-9574-02d25a20f331-plugins-config-volume\") pod \"95aee87a-8056-47a5-9574-02d25a20f331\" (UID: \"95aee87a-8056-47a5-9574-02d25a20f331\") " Nov 03 21:18:16.656310 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:16.656289 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95aee87a-8056-47a5-9574-02d25a20f331-kube-api-access-qkcj5\" (UniqueName: \"kubernetes.io/projected/95aee87a-8056-47a5-9574-02d25a20f331-kube-api-access-qkcj5\") pod \"95aee87a-8056-47a5-9574-02d25a20f331\" (UID: \"95aee87a-8056-47a5-9574-02d25a20f331\") " Nov 03 21:18:16.656560 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:16.656528 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95aee87a-8056-47a5-9574-02d25a20f331-plugins-config-volume" pod "95aee87a-8056-47a5-9574-02d25a20f331" (UID: "95aee87a-8056-47a5-9574-02d25a20f331"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:18:16.656971 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:16.656948 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95aee87a-8056-47a5-9574-02d25a20f331-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:16.658564 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:16.658535 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95aee87a-8056-47a5-9574-02d25a20f331-kube-api-access-qkcj5" pod "95aee87a-8056-47a5-9574-02d25a20f331" (UID: "95aee87a-8056-47a5-9574-02d25a20f331"). InnerVolumeSpecName "kube-api-access-qkcj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:18:16.780116 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:16.776727 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkcj5\" (UniqueName: \"kubernetes.io/projected/95aee87a-8056-47a5-9574-02d25a20f331-kube-api-access-qkcj5\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:17.143086 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:17.142985 5615 generic.go:356] "Generic (PLEG): container finished" podID="95aee87a-8056-47a5-9574-02d25a20f331" containerID="02e2a94d8703d085d7951fd65c8539de2e9a80a69b928c32c347318d48b355fc" exitCode=0 Nov 03 21:18:17.143086 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:17.143038 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-78b68f9566-6hnx6" Nov 03 21:18:17.143612 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:17.143090 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-78b68f9566-6hnx6" event={"ID":"95aee87a-8056-47a5-9574-02d25a20f331","Type":"ContainerDied","Data":"02e2a94d8703d085d7951fd65c8539de2e9a80a69b928c32c347318d48b355fc"} Nov 03 21:18:17.143612 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:17.143149 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-78b68f9566-6hnx6" event={"ID":"95aee87a-8056-47a5-9574-02d25a20f331","Type":"ContainerDied","Data":"d5f111a5a11dff8773b476fb5985cb980e5102359c90d211bf779c1c5e739b11"} Nov 03 21:18:17.143612 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:17.143180 5615 scope.go:122] "RemoveContainer" containerID="02e2a94d8703d085d7951fd65c8539de2e9a80a69b928c32c347318d48b355fc" Nov 03 21:18:17.144691 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:17.144671 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm" Nov 03 21:18:17.144769 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:17.144673 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm" event={"ID":"c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867","Type":"ContainerDied","Data":"57a74129ae2a6b02f1a96503a645811784301f1e597b55ffde8546a646b6ddce"} Nov 03 21:18:17.157349 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:17.157255 5615 scope.go:122] "RemoveContainer" containerID="02e2a94d8703d085d7951fd65c8539de2e9a80a69b928c32c347318d48b355fc" Nov 03 21:18:17.157684 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:18:17.157660 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02e2a94d8703d085d7951fd65c8539de2e9a80a69b928c32c347318d48b355fc\": container with ID starting with 02e2a94d8703d085d7951fd65c8539de2e9a80a69b928c32c347318d48b355fc not found: ID does not exist" containerID="02e2a94d8703d085d7951fd65c8539de2e9a80a69b928c32c347318d48b355fc" Nov 03 21:18:17.157783 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:17.157695 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02e2a94d8703d085d7951fd65c8539de2e9a80a69b928c32c347318d48b355fc"} err="failed to get container status \"02e2a94d8703d085d7951fd65c8539de2e9a80a69b928c32c347318d48b355fc\": rpc error: code = NotFound desc = could not find container \"02e2a94d8703d085d7951fd65c8539de2e9a80a69b928c32c347318d48b355fc\": container with ID starting with 02e2a94d8703d085d7951fd65c8539de2e9a80a69b928c32c347318d48b355fc not found: ID does not exist" Nov 03 21:18:17.157783 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:17.157724 5615 scope.go:122] "RemoveContainer" containerID="5c6f5dcac0507028db1510fd254d52ff4022dc6bc988b077f5fa3419bbd90d31" Nov 03 21:18:17.301396 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:17.301359 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-78b68f9566-6hnx6"] Nov 03 21:18:17.307017 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:17.306985 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0f8726-78b68f9566-6hnx6"] Nov 03 21:18:17.419033 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:17.418935 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm"] Nov 03 21:18:17.419486 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:17.419456 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0f8726-544457c4f7-pvdfm"] Nov 03 21:18:17.726145 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:17.725185 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95aee87a-8056-47a5-9574-02d25a20f331" path="/var/lib/kubelet/pods/95aee87a-8056-47a5-9574-02d25a20f331/volumes" Nov 03 21:18:17.726322 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:17.726212 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867" path="/var/lib/kubelet/pods/c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867/volumes" Nov 03 21:18:17.986190 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:17.986162 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-7765fcd666-6l4wb" Nov 03 21:18:31.761044 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:31.761005 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k"] Nov 03 21:18:31.761483 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:31.761281 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k" podUID="4539a85b-3f5b-40e1-9caa-9407bf616dc8" containerName="maas-api" containerID="cri-o://2a36ca7d8124bb12bda5b8dbe505174f4caef277afc7f5b17773112434b8bfbb" gracePeriod=30 Nov 03 21:18:32.277734 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:32.277665 5615 generic.go:356] "Generic (PLEG): container finished" podID="4539a85b-3f5b-40e1-9caa-9407bf616dc8" containerID="2a36ca7d8124bb12bda5b8dbe505174f4caef277afc7f5b17773112434b8bfbb" exitCode=0 Nov 03 21:18:32.278203 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:32.277863 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k" event={"ID":"4539a85b-3f5b-40e1-9caa-9407bf616dc8","Type":"ContainerDied","Data":"2a36ca7d8124bb12bda5b8dbe505174f4caef277afc7f5b17773112434b8bfbb"} Nov 03 21:18:32.278464 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:32.278439 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k" event={"ID":"4539a85b-3f5b-40e1-9caa-9407bf616dc8","Type":"ContainerDied","Data":"be3518b69e7cfec249faa1286a118d78fa9a55148a482780a0463877525fb822"} Nov 03 21:18:32.278808 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:32.278779 5615 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be3518b69e7cfec249faa1286a118d78fa9a55148a482780a0463877525fb822" Nov 03 21:18:32.282323 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:32.282303 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k" Nov 03 21:18:32.346102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:32.340086 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4539a85b-3f5b-40e1-9caa-9407bf616dc8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4539a85b-3f5b-40e1-9caa-9407bf616dc8-maas-api-tls\") pod \"4539a85b-3f5b-40e1-9caa-9407bf616dc8\" (UID: \"4539a85b-3f5b-40e1-9caa-9407bf616dc8\") " Nov 03 21:18:32.346102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:32.340232 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4539a85b-3f5b-40e1-9caa-9407bf616dc8-kube-api-access-bq4lk\" (UniqueName: \"kubernetes.io/projected/4539a85b-3f5b-40e1-9caa-9407bf616dc8-kube-api-access-bq4lk\") pod \"4539a85b-3f5b-40e1-9caa-9407bf616dc8\" (UID: \"4539a85b-3f5b-40e1-9caa-9407bf616dc8\") " Nov 03 21:18:32.346102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:32.340334 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4539a85b-3f5b-40e1-9caa-9407bf616dc8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4539a85b-3f5b-40e1-9caa-9407bf616dc8-metrics-certs\") pod \"4539a85b-3f5b-40e1-9caa-9407bf616dc8\" (UID: \"4539a85b-3f5b-40e1-9caa-9407bf616dc8\") " Nov 03 21:18:32.346102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:32.340488 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4539a85b-3f5b-40e1-9caa-9407bf616dc8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4539a85b-3f5b-40e1-9caa-9407bf616dc8-openshift-service-ca\") pod \"4539a85b-3f5b-40e1-9caa-9407bf616dc8\" (UID: \"4539a85b-3f5b-40e1-9caa-9407bf616dc8\") " Nov 03 21:18:32.346102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:32.341646 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4539a85b-3f5b-40e1-9caa-9407bf616dc8-openshift-service-ca" pod "4539a85b-3f5b-40e1-9caa-9407bf616dc8" (UID: "4539a85b-3f5b-40e1-9caa-9407bf616dc8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:18:32.369727 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:32.369205 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4539a85b-3f5b-40e1-9caa-9407bf616dc8-metrics-certs" pod "4539a85b-3f5b-40e1-9caa-9407bf616dc8" (UID: "4539a85b-3f5b-40e1-9caa-9407bf616dc8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:18:32.381582 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:32.372249 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4539a85b-3f5b-40e1-9caa-9407bf616dc8-maas-api-tls" pod "4539a85b-3f5b-40e1-9caa-9407bf616dc8" (UID: "4539a85b-3f5b-40e1-9caa-9407bf616dc8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:18:32.381582 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:32.372668 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4539a85b-3f5b-40e1-9caa-9407bf616dc8-kube-api-access-bq4lk" pod "4539a85b-3f5b-40e1-9caa-9407bf616dc8" (UID: "4539a85b-3f5b-40e1-9caa-9407bf616dc8"). InnerVolumeSpecName "kube-api-access-bq4lk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:18:32.442483 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:32.442445 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4539a85b-3f5b-40e1-9caa-9407bf616dc8-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:32.442483 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:32.442480 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4539a85b-3f5b-40e1-9caa-9407bf616dc8-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:32.442483 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:32.442490 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq4lk\" (UniqueName: \"kubernetes.io/projected/4539a85b-3f5b-40e1-9caa-9407bf616dc8-kube-api-access-bq4lk\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:32.442781 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:32.442499 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4539a85b-3f5b-40e1-9caa-9407bf616dc8-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:33.285387 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:33.285353 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k" Nov 03 21:18:33.442734 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:33.442692 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k"] Nov 03 21:18:33.523589 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:33.523549 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9232e5-6fd7cbdfcc-5fj2k"] Nov 03 21:18:33.721148 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:33.716236 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4539a85b-3f5b-40e1-9caa-9407bf616dc8" path="/var/lib/kubelet/pods/4539a85b-3f5b-40e1-9caa-9407bf616dc8/volumes" Nov 03 21:18:39.223287 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:39.223149 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc"] Nov 03 21:18:39.223794 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:39.223521 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" podUID="1310ac80-c9fa-4498-8bad-6be0ded4c495" containerName="istio-proxy" containerID="cri-o://62ee231e6df2e92b8acfa11f39aa326fe018a16ad0013d51e9a2752b34f2ae36" gracePeriod=30 Nov 03 21:18:40.620859 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:18:40.620818 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:18:42.469294 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:42.469241 5615 patch_prober.go:29] interesting pod/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:18:42.469899 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:42.469326 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" podUID="f82eafd0-9398-4cec-9b95-e510391864ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:18:44.444101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.442662 5615 generic.go:356] "Generic (PLEG): container finished" podID="1310ac80-c9fa-4498-8bad-6be0ded4c495" containerID="62ee231e6df2e92b8acfa11f39aa326fe018a16ad0013d51e9a2752b34f2ae36" exitCode=0 Nov 03 21:18:44.444101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.442920 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" event={"ID":"1310ac80-c9fa-4498-8bad-6be0ded4c495","Type":"ContainerDied","Data":"62ee231e6df2e92b8acfa11f39aa326fe018a16ad0013d51e9a2752b34f2ae36"} Nov 03 21:18:44.444101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.442987 5615 scope.go:122] "RemoveContainer" containerID="dc18fcaae9950171f61627ff17264f9ac5dd80f3e63812ec6c525f146a3b9a08" Nov 03 21:18:44.447256 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.447222 5615 generic.go:356] "Generic (PLEG): container finished" podID="f82eafd0-9398-4cec-9b95-e510391864ab" containerID="efc7d29bc549dd372ddcb388afb5e8cadc82bf8e97142c1f562db32ff49fe629" exitCode=0 Nov 03 21:18:44.447377 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.447308 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" event={"ID":"f82eafd0-9398-4cec-9b95-e510391864ab","Type":"ContainerDied","Data":"efc7d29bc549dd372ddcb388afb5e8cadc82bf8e97142c1f562db32ff49fe629"} Nov 03 21:18:44.447844 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.447819 5615 scope.go:122] "RemoveContainer" containerID="efc7d29bc549dd372ddcb388afb5e8cadc82bf8e97142c1f562db32ff49fe629" Nov 03 21:18:44.449864 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.449826 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:18:44.449976 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.449934 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:18:44.450044 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.449985 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:18:44.555747 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.555719 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:18:44.630994 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.630936 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1310ac80-c9fa-4498-8bad-6be0ded4c495-kube-api-access-s82bm\" (UniqueName: \"kubernetes.io/projected/1310ac80-c9fa-4498-8bad-6be0ded4c495-kube-api-access-s82bm\") pod \"1310ac80-c9fa-4498-8bad-6be0ded4c495\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " Nov 03 21:18:44.631218 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.631160 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-workload-certs\") pod \"1310ac80-c9fa-4498-8bad-6be0ded4c495\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " Nov 03 21:18:44.631297 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.631256 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-istio-data\") pod \"1310ac80-c9fa-4498-8bad-6be0ded4c495\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " Nov 03 21:18:44.631388 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.631316 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-workload-socket\") pod \"1310ac80-c9fa-4498-8bad-6be0ded4c495\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " Nov 03 21:18:44.631479 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.631387 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1310ac80-c9fa-4498-8bad-6be0ded4c495-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1310ac80-c9fa-4498-8bad-6be0ded4c495-istiod-ca-cert\") pod \"1310ac80-c9fa-4498-8bad-6be0ded4c495\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " Nov 03 21:18:44.631479 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.631442 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1310ac80-c9fa-4498-8bad-6be0ded4c495-istio-token\" (UniqueName: \"kubernetes.io/projected/1310ac80-c9fa-4498-8bad-6be0ded4c495-istio-token\") pod \"1310ac80-c9fa-4498-8bad-6be0ded4c495\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " Nov 03 21:18:44.631619 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.631535 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-istio-envoy\") pod \"1310ac80-c9fa-4498-8bad-6be0ded4c495\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " Nov 03 21:18:44.631619 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.631586 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-credential-socket\") pod \"1310ac80-c9fa-4498-8bad-6be0ded4c495\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " Nov 03 21:18:44.631783 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.631627 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1310ac80-c9fa-4498-8bad-6be0ded4c495-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1310ac80-c9fa-4498-8bad-6be0ded4c495-istio-podinfo\") pod \"1310ac80-c9fa-4498-8bad-6be0ded4c495\" (UID: \"1310ac80-c9fa-4498-8bad-6be0ded4c495\") " Nov 03 21:18:44.633049 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.632908 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-workload-socket" pod "1310ac80-c9fa-4498-8bad-6be0ded4c495" (UID: "1310ac80-c9fa-4498-8bad-6be0ded4c495"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:44.633481 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.633451 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-workload-certs" pod "1310ac80-c9fa-4498-8bad-6be0ded4c495" (UID: "1310ac80-c9fa-4498-8bad-6be0ded4c495"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:44.633683 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.633659 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-istio-data" pod "1310ac80-c9fa-4498-8bad-6be0ded4c495" (UID: "1310ac80-c9fa-4498-8bad-6be0ded4c495"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:44.633990 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.633955 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1310ac80-c9fa-4498-8bad-6be0ded4c495-istiod-ca-cert" pod "1310ac80-c9fa-4498-8bad-6be0ded4c495" (UID: "1310ac80-c9fa-4498-8bad-6be0ded4c495"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:18:44.634878 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.634849 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1310ac80-c9fa-4498-8bad-6be0ded4c495-istio-podinfo" pod "1310ac80-c9fa-4498-8bad-6be0ded4c495" (UID: "1310ac80-c9fa-4498-8bad-6be0ded4c495"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:18:44.635238 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.635211 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-credential-socket" pod "1310ac80-c9fa-4498-8bad-6be0ded4c495" (UID: "1310ac80-c9fa-4498-8bad-6be0ded4c495"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:44.636631 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.636603 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1310ac80-c9fa-4498-8bad-6be0ded4c495-kube-api-access-s82bm" pod "1310ac80-c9fa-4498-8bad-6be0ded4c495" (UID: "1310ac80-c9fa-4498-8bad-6be0ded4c495"). InnerVolumeSpecName "kube-api-access-s82bm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:18:44.637196 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.637174 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-istio-envoy" pod "1310ac80-c9fa-4498-8bad-6be0ded4c495" (UID: "1310ac80-c9fa-4498-8bad-6be0ded4c495"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:44.637980 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.637955 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1310ac80-c9fa-4498-8bad-6be0ded4c495-istio-token" pod "1310ac80-c9fa-4498-8bad-6be0ded4c495" (UID: "1310ac80-c9fa-4498-8bad-6be0ded4c495"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:18:44.735522 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.735022 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s82bm\" (UniqueName: \"kubernetes.io/projected/1310ac80-c9fa-4498-8bad-6be0ded4c495-kube-api-access-s82bm\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:44.735522 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.735061 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:44.735522 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.735090 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:44.735522 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.735104 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:44.735522 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.735119 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1310ac80-c9fa-4498-8bad-6be0ded4c495-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:44.735522 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.735131 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1310ac80-c9fa-4498-8bad-6be0ded4c495-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:44.735522 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.735144 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:44.735522 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.735157 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1310ac80-c9fa-4498-8bad-6be0ded4c495-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:44.735522 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:44.735170 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1310ac80-c9fa-4498-8bad-6be0ded4c495-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:45.455011 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:45.454978 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" Nov 03 21:18:45.455483 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:45.455025 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc" event={"ID":"1310ac80-c9fa-4498-8bad-6be0ded4c495","Type":"ContainerDied","Data":"ef256ae459c49c71051a58723abc052ac912ca205589605aa7daa817ca1da15c"} Nov 03 21:18:45.455483 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:45.455095 5615 scope.go:122] "RemoveContainer" containerID="62ee231e6df2e92b8acfa11f39aa326fe018a16ad0013d51e9a2752b34f2ae36" Nov 03 21:18:45.457790 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:45.457763 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" event={"ID":"f82eafd0-9398-4cec-9b95-e510391864ab","Type":"ContainerStarted","Data":"bc73b1363a507690decd4d5ae2d0c21653d49b7b199a7a0b3d31c4e4a9518b67"} Nov 03 21:18:45.672536 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:45.672497 5615 patch_prober.go:29] interesting pod/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 21:18:45.674967 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:45.674925 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" podUID="f82eafd0-9398-4cec-9b95-e510391864ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 21:18:45.684089 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:45.684042 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:18:45.769391 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:45.769348 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc"] Nov 03 21:18:45.776138 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:45.776099 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-9232e5-openshift-default-694b9d78d9-ssgxc"] Nov 03 21:18:46.651333 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:46.651297 5615 patch_prober.go:29] interesting pod/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 21:18:46.651740 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:46.651362 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" podUID="f82eafd0-9398-4cec-9b95-e510391864ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 21:18:47.659987 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:47.659950 5615 patch_prober.go:29] interesting pod/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 21:18:47.660689 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:47.660639 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" podUID="f82eafd0-9398-4cec-9b95-e510391864ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 21:18:47.666130 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:47.666098 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1310ac80-c9fa-4498-8bad-6be0ded4c495" path="/var/lib/kubelet/pods/1310ac80-c9fa-4498-8bad-6be0ded4c495/volumes" Nov 03 21:18:48.651758 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:48.651720 5615 patch_prober.go:29] interesting pod/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 21:18:48.651936 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:48.651791 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" podUID="f82eafd0-9398-4cec-9b95-e510391864ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 21:18:49.520607 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.520563 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh"] Nov 03 21:18:49.530235 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.530205 5615 cpu_manager.go:414] "containerMap: removing container" podUID="c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867" containerName="maas-api" Nov 03 21:18:49.530538 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.530521 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867" containerName="maas-api" Nov 03 21:18:49.530679 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.530668 5615 cpu_manager.go:414] "containerMap: removing container" podUID="4539a85b-3f5b-40e1-9caa-9407bf616dc8" containerName="maas-api" Nov 03 21:18:49.530748 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.530739 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4539a85b-3f5b-40e1-9caa-9407bf616dc8" containerName="maas-api" Nov 03 21:18:49.530839 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.530831 5615 cpu_manager.go:414] "containerMap: removing container" podUID="1310ac80-c9fa-4498-8bad-6be0ded4c495" containerName="istio-proxy" Nov 03 21:18:49.530907 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.530898 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1310ac80-c9fa-4498-8bad-6be0ded4c495" containerName="istio-proxy" Nov 03 21:18:49.530979 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.530971 5615 cpu_manager.go:414] "containerMap: removing container" podUID="95aee87a-8056-47a5-9574-02d25a20f331" containerName="payload-pre-processing" Nov 03 21:18:49.531110 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.531030 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95aee87a-8056-47a5-9574-02d25a20f331" containerName="payload-pre-processing" Nov 03 21:18:49.531421 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.531403 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="4539a85b-3f5b-40e1-9caa-9407bf616dc8" containerName="maas-api" Nov 03 21:18:49.531545 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.531533 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="1310ac80-c9fa-4498-8bad-6be0ded4c495" containerName="istio-proxy" Nov 03 21:18:49.543211 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.543179 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="95aee87a-8056-47a5-9574-02d25a20f331" containerName="payload-pre-processing" Nov 03 21:18:49.543460 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.543445 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="1310ac80-c9fa-4498-8bad-6be0ded4c495" containerName="istio-proxy" Nov 03 21:18:49.543556 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.543541 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="c51c9d1c-e918-4a50-a3b7-4eb7f1e2f867" containerName="maas-api" Nov 03 21:18:49.552426 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.552395 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh" Nov 03 21:18:49.570417 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.564524 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh"] Nov 03 21:18:49.570417 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.569492 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae01697d-2a38-41f0-8bcf-695b6ec9d967-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh\" (UID: \"ae01697d-2a38-41f0-8bcf-695b6ec9d967\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh" Nov 03 21:18:49.570417 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.570116 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7259\" (UniqueName: \"kubernetes.io/projected/ae01697d-2a38-41f0-8bcf-695b6ec9d967-kube-api-access-s7259\") pod \"payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh\" (UID: \"ae01697d-2a38-41f0-8bcf-695b6ec9d967\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh" Nov 03 21:18:49.650955 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.650919 5615 patch_prober.go:29] interesting pod/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 21:18:49.651399 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.651362 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" podUID="f82eafd0-9398-4cec-9b95-e510391864ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 21:18:49.652554 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.652534 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-3e1b49\"" Nov 03 21:18:49.658030 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.658002 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-3e1b49-dockercfg-jbpkn\"" Nov 03 21:18:49.680819 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.680773 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae01697d-2a38-41f0-8bcf-695b6ec9d967-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh\" (UID: \"ae01697d-2a38-41f0-8bcf-695b6ec9d967\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh" Nov 03 21:18:49.681190 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.681150 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7259\" (UniqueName: \"kubernetes.io/projected/ae01697d-2a38-41f0-8bcf-695b6ec9d967-kube-api-access-s7259\") pod \"payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh\" (UID: \"ae01697d-2a38-41f0-8bcf-695b6ec9d967\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh" Nov 03 21:18:49.682847 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.682821 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae01697d-2a38-41f0-8bcf-695b6ec9d967-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh\" (UID: \"ae01697d-2a38-41f0-8bcf-695b6ec9d967\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh" Nov 03 21:18:49.796146 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.792839 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr"] Nov 03 21:18:49.796146 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.793943 5615 cpu_manager.go:414] "containerMap: removing container" podUID="1310ac80-c9fa-4498-8bad-6be0ded4c495" containerName="istio-proxy" Nov 03 21:18:49.796146 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.793966 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1310ac80-c9fa-4498-8bad-6be0ded4c495" containerName="istio-proxy" Nov 03 21:18:49.833557 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.833477 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7259\" (UniqueName: \"kubernetes.io/projected/ae01697d-2a38-41f0-8bcf-695b6ec9d967-kube-api-access-s7259\") pod \"payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh\" (UID: \"ae01697d-2a38-41f0-8bcf-695b6ec9d967\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh" Nov 03 21:18:49.873065 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.873017 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr"] Nov 03 21:18:49.873305 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.873179 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr" Nov 03 21:18:49.891992 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.891941 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh" Nov 03 21:18:49.950738 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.950699 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a23823d8-ee6f-4b00-879c-0c05498338ae-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr\" (UID: \"a23823d8-ee6f-4b00-879c-0c05498338ae\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr" Nov 03 21:18:49.950910 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:49.950839 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnjjb\" (UniqueName: \"kubernetes.io/projected/a23823d8-ee6f-4b00-879c-0c05498338ae-kube-api-access-fnjjb\") pod \"payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr\" (UID: \"a23823d8-ee6f-4b00-879c-0c05498338ae\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr" Nov 03 21:18:50.051796 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:50.051710 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a23823d8-ee6f-4b00-879c-0c05498338ae-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr\" (UID: \"a23823d8-ee6f-4b00-879c-0c05498338ae\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr" Nov 03 21:18:50.051942 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:50.051823 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnjjb\" (UniqueName: \"kubernetes.io/projected/a23823d8-ee6f-4b00-879c-0c05498338ae-kube-api-access-fnjjb\") pod \"payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr\" (UID: \"a23823d8-ee6f-4b00-879c-0c05498338ae\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr" Nov 03 21:18:50.052593 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:50.052570 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a23823d8-ee6f-4b00-879c-0c05498338ae-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr\" (UID: \"a23823d8-ee6f-4b00-879c-0c05498338ae\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr" Nov 03 21:18:50.128101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:50.128039 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnjjb\" (UniqueName: \"kubernetes.io/projected/a23823d8-ee6f-4b00-879c-0c05498338ae-kube-api-access-fnjjb\") pod \"payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr\" (UID: \"a23823d8-ee6f-4b00-879c-0c05498338ae\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr" Nov 03 21:18:50.196470 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:50.196432 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr" Nov 03 21:18:50.328734 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:50.324768 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh"] Nov 03 21:18:50.333580 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:18:50.333532 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae01697d_2a38_41f0_8bcf_695b6ec9d967.slice/crio-0d3ca6acfec4421b92273505a316435c1504bd24a79fd10b3c322c84b401eb74 WatchSource:0}: Error finding container 0d3ca6acfec4421b92273505a316435c1504bd24a79fd10b3c322c84b401eb74: Status 404 returned error can't find the container with id 0d3ca6acfec4421b92273505a316435c1504bd24a79fd10b3c322c84b401eb74 Nov 03 21:18:50.530581 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:50.530536 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh" event={"ID":"ae01697d-2a38-41f0-8bcf-695b6ec9d967","Type":"ContainerStarted","Data":"81c60eb74f04a1b930c5b5d7cbbf4065db676651cfe27b773e8e8430982178ca"} Nov 03 21:18:50.530970 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:50.530594 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh" event={"ID":"ae01697d-2a38-41f0-8bcf-695b6ec9d967","Type":"ContainerStarted","Data":"0d3ca6acfec4421b92273505a316435c1504bd24a79fd10b3c322c84b401eb74"} Nov 03 21:18:50.531444 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:50.531423 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh" Nov 03 21:18:50.641782 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:50.641669 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh" podStartSLOduration=1.641657154 podStartE2EDuration="1.641657154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:18:50.641495669 +0000 UTC m=+1825.109274393" watchObservedRunningTime="2026-11-03 21:18:50.641657154 +0000 UTC m=+1825.109435876" Nov 03 21:18:50.651396 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:50.651359 5615 patch_prober.go:29] interesting pod/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 21:18:50.651595 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:50.651418 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" podUID="f82eafd0-9398-4cec-9b95-e510391864ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 21:18:50.681496 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:50.681469 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr"] Nov 03 21:18:50.683123 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:18:50.683087 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda23823d8_ee6f_4b00_879c_0c05498338ae.slice/crio-535d373db9c6c254b9f424cf353007da83e806ae6f319cc489b7a16c1328ce03 WatchSource:0}: Error finding container 535d373db9c6c254b9f424cf353007da83e806ae6f319cc489b7a16c1328ce03: Status 404 returned error can't find the container with id 535d373db9c6c254b9f424cf353007da83e806ae6f319cc489b7a16c1328ce03 Nov 03 21:18:51.538566 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:51.538517 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr" event={"ID":"a23823d8-ee6f-4b00-879c-0c05498338ae","Type":"ContainerStarted","Data":"3f688a139a79ea764546782fcd995e944067aa33708eb8013f374dc1d4720ef0"} Nov 03 21:18:51.538566 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:51.538574 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr" event={"ID":"a23823d8-ee6f-4b00-879c-0c05498338ae","Type":"ContainerStarted","Data":"535d373db9c6c254b9f424cf353007da83e806ae6f319cc489b7a16c1328ce03"} Nov 03 21:18:51.539057 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:51.538840 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr" Nov 03 21:18:51.649854 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:51.649795 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr" podStartSLOduration=2.64977902 podStartE2EDuration="2.64977902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:18:51.648450833 +0000 UTC m=+1826.116229561" watchObservedRunningTime="2026-11-03 21:18:51.64977902 +0000 UTC m=+1826.117557742" Nov 03 21:18:51.655044 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:51.655012 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:18:51.656325 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:51.656294 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:18:51.660914 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:51.660887 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:18:54.093814 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:54.093776 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg"] Nov 03 21:18:54.094634 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:54.094601 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg" podUID="6c156401-fe0c-4439-b1a8-7c9bdd23a624" containerName="maas-api" containerID="cri-o://f2cc3d4302d9e7cfa7609348651f78e81005812a5d1c3f34dc1f9c744ec2ee24" gracePeriod=30 Nov 03 21:18:54.558804 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:54.558772 5615 generic.go:356] "Generic (PLEG): container finished" podID="6c156401-fe0c-4439-b1a8-7c9bdd23a624" containerID="f2cc3d4302d9e7cfa7609348651f78e81005812a5d1c3f34dc1f9c744ec2ee24" exitCode=0 Nov 03 21:18:54.559017 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:54.558944 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg" event={"ID":"6c156401-fe0c-4439-b1a8-7c9bdd23a624","Type":"ContainerDied","Data":"f2cc3d4302d9e7cfa7609348651f78e81005812a5d1c3f34dc1f9c744ec2ee24"} Nov 03 21:18:54.561997 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:54.559855 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg" event={"ID":"6c156401-fe0c-4439-b1a8-7c9bdd23a624","Type":"ContainerDied","Data":"2c70d9b09240279dcf601802d92c500a482905a2e5aaec8d444cccfebbb11dcc"} Nov 03 21:18:54.594115 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:54.585508 5615 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c70d9b09240279dcf601802d92c500a482905a2e5aaec8d444cccfebbb11dcc" Nov 03 21:18:54.646113 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:54.624679 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg" Nov 03 21:18:54.693381 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:54.693339 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c156401-fe0c-4439-b1a8-7c9bdd23a624-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c156401-fe0c-4439-b1a8-7c9bdd23a624-metrics-certs\") pod \"6c156401-fe0c-4439-b1a8-7c9bdd23a624\" (UID: \"6c156401-fe0c-4439-b1a8-7c9bdd23a624\") " Nov 03 21:18:54.693596 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:54.693451 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c156401-fe0c-4439-b1a8-7c9bdd23a624-kube-api-access-kphjf\" (UniqueName: \"kubernetes.io/projected/6c156401-fe0c-4439-b1a8-7c9bdd23a624-kube-api-access-kphjf\") pod \"6c156401-fe0c-4439-b1a8-7c9bdd23a624\" (UID: \"6c156401-fe0c-4439-b1a8-7c9bdd23a624\") " Nov 03 21:18:54.693596 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:54.693499 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c156401-fe0c-4439-b1a8-7c9bdd23a624-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c156401-fe0c-4439-b1a8-7c9bdd23a624-maas-api-tls\") pod \"6c156401-fe0c-4439-b1a8-7c9bdd23a624\" (UID: \"6c156401-fe0c-4439-b1a8-7c9bdd23a624\") " Nov 03 21:18:54.693596 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:54.693526 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c156401-fe0c-4439-b1a8-7c9bdd23a624-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c156401-fe0c-4439-b1a8-7c9bdd23a624-openshift-service-ca\") pod \"6c156401-fe0c-4439-b1a8-7c9bdd23a624\" (UID: \"6c156401-fe0c-4439-b1a8-7c9bdd23a624\") " Nov 03 21:18:54.697112 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:54.696744 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c156401-fe0c-4439-b1a8-7c9bdd23a624-openshift-service-ca" pod "6c156401-fe0c-4439-b1a8-7c9bdd23a624" (UID: "6c156401-fe0c-4439-b1a8-7c9bdd23a624"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:18:54.701107 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:54.699181 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c156401-fe0c-4439-b1a8-7c9bdd23a624-metrics-certs" pod "6c156401-fe0c-4439-b1a8-7c9bdd23a624" (UID: "6c156401-fe0c-4439-b1a8-7c9bdd23a624"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:18:54.704287 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:54.704255 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c156401-fe0c-4439-b1a8-7c9bdd23a624-maas-api-tls" pod "6c156401-fe0c-4439-b1a8-7c9bdd23a624" (UID: "6c156401-fe0c-4439-b1a8-7c9bdd23a624"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:18:54.706275 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:54.706236 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c156401-fe0c-4439-b1a8-7c9bdd23a624-kube-api-access-kphjf" pod "6c156401-fe0c-4439-b1a8-7c9bdd23a624" (UID: "6c156401-fe0c-4439-b1a8-7c9bdd23a624"). InnerVolumeSpecName "kube-api-access-kphjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:18:54.795875 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:54.795839 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kphjf\" (UniqueName: \"kubernetes.io/projected/6c156401-fe0c-4439-b1a8-7c9bdd23a624-kube-api-access-kphjf\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:54.795875 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:54.795871 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c156401-fe0c-4439-b1a8-7c9bdd23a624-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:54.795875 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:54.795882 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c156401-fe0c-4439-b1a8-7c9bdd23a624-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:54.796146 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:54.795891 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c156401-fe0c-4439-b1a8-7c9bdd23a624-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:55.564996 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:55.564962 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg" Nov 03 21:18:55.767780 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:55.765984 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg"] Nov 03 21:18:55.791610 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:55.791577 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-32f6e43c-c95f97ffd-5p6jg"] Nov 03 21:18:56.349583 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:56.349542 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np"] Nov 03 21:18:56.349872 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:56.349842 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" podUID="c39084a3-cdf2-47b8-8aa1-b09415e6a90e" containerName="istio-proxy" containerID="cri-o://a0d42e39b268b810fa13e640e811f72567868c8f330fb2d2d3c1f286792680b1" gracePeriod=30 Nov 03 21:18:57.652885 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:18:57.652848 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c156401-fe0c-4439-b1a8-7c9bdd23a624" path="/var/lib/kubelet/pods/6c156401-fe0c-4439-b1a8-7c9bdd23a624/volumes" Nov 03 21:19:01.629365 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.629288 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:19:01.633549 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.633509 5615 generic.go:356] "Generic (PLEG): container finished" podID="c39084a3-cdf2-47b8-8aa1-b09415e6a90e" containerID="a0d42e39b268b810fa13e640e811f72567868c8f330fb2d2d3c1f286792680b1" exitCode=0 Nov 03 21:19:01.633689 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.633568 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" Nov 03 21:19:01.633689 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.633589 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" event={"ID":"c39084a3-cdf2-47b8-8aa1-b09415e6a90e","Type":"ContainerDied","Data":"a0d42e39b268b810fa13e640e811f72567868c8f330fb2d2d3c1f286792680b1"} Nov 03 21:19:01.633689 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.633666 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np" event={"ID":"c39084a3-cdf2-47b8-8aa1-b09415e6a90e","Type":"ContainerDied","Data":"ef8397f2a820de0bc654271e9e86c9b4d09387d215b1f33d941ee7bd7a7883ca"} Nov 03 21:19:01.633689 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.633689 5615 scope.go:122] "RemoveContainer" containerID="a0d42e39b268b810fa13e640e811f72567868c8f330fb2d2d3c1f286792680b1" Nov 03 21:19:01.635593 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.635571 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-79d96df5c6-cfzcv"] Nov 03 21:19:01.643694 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.643672 5615 scope.go:122] "RemoveContainer" containerID="31513fd42aea31fc8770bcab3d3bb3b094e6646b731484d0fb5364935f52ea3b" Nov 03 21:19:01.654155 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.654129 5615 scope.go:122] "RemoveContainer" containerID="a0d42e39b268b810fa13e640e811f72567868c8f330fb2d2d3c1f286792680b1" Nov 03 21:19:01.654461 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:19:01.654432 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0d42e39b268b810fa13e640e811f72567868c8f330fb2d2d3c1f286792680b1\": container with ID starting with a0d42e39b268b810fa13e640e811f72567868c8f330fb2d2d3c1f286792680b1 not found: ID does not exist" containerID="a0d42e39b268b810fa13e640e811f72567868c8f330fb2d2d3c1f286792680b1" Nov 03 21:19:01.654534 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.654466 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0d42e39b268b810fa13e640e811f72567868c8f330fb2d2d3c1f286792680b1"} err="failed to get container status \"a0d42e39b268b810fa13e640e811f72567868c8f330fb2d2d3c1f286792680b1\": rpc error: code = NotFound desc = could not find container \"a0d42e39b268b810fa13e640e811f72567868c8f330fb2d2d3c1f286792680b1\": container with ID starting with a0d42e39b268b810fa13e640e811f72567868c8f330fb2d2d3c1f286792680b1 not found: ID does not exist" Nov 03 21:19:01.654534 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.654485 5615 scope.go:122] "RemoveContainer" containerID="31513fd42aea31fc8770bcab3d3bb3b094e6646b731484d0fb5364935f52ea3b" Nov 03 21:19:01.654763 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:19:01.654734 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31513fd42aea31fc8770bcab3d3bb3b094e6646b731484d0fb5364935f52ea3b\": container with ID starting with 31513fd42aea31fc8770bcab3d3bb3b094e6646b731484d0fb5364935f52ea3b not found: ID does not exist" containerID="31513fd42aea31fc8770bcab3d3bb3b094e6646b731484d0fb5364935f52ea3b" Nov 03 21:19:01.654822 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.654769 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31513fd42aea31fc8770bcab3d3bb3b094e6646b731484d0fb5364935f52ea3b"} err="failed to get container status \"31513fd42aea31fc8770bcab3d3bb3b094e6646b731484d0fb5364935f52ea3b\": rpc error: code = NotFound desc = could not find container \"31513fd42aea31fc8770bcab3d3bb3b094e6646b731484d0fb5364935f52ea3b\": container with ID starting with 31513fd42aea31fc8770bcab3d3bb3b094e6646b731484d0fb5364935f52ea3b not found: ID does not exist" Nov 03 21:19:01.740667 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.740630 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istio-data\") pod \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " Nov 03 21:19:01.740866 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.740688 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istio-token\" (UniqueName: \"kubernetes.io/projected/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istio-token\") pod \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " Nov 03 21:19:01.740866 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.740757 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-credential-socket\") pod \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " Nov 03 21:19:01.740866 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.740798 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-workload-socket\") pod \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " Nov 03 21:19:01.740866 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.740824 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istio-envoy\") pod \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " Nov 03 21:19:01.741132 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.740886 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-workload-certs\") pod \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " Nov 03 21:19:01.741132 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.740954 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istio-podinfo\") pod \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " Nov 03 21:19:01.741132 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.740986 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istio-data" pod "c39084a3-cdf2-47b8-8aa1-b09415e6a90e" (UID: "c39084a3-cdf2-47b8-8aa1-b09415e6a90e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:01.741132 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.741016 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istiod-ca-cert\") pod \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " Nov 03 21:19:01.741132 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.741049 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-kube-api-access-v9dwq\" (UniqueName: \"kubernetes.io/projected/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-kube-api-access-v9dwq\") pod \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\" (UID: \"c39084a3-cdf2-47b8-8aa1-b09415e6a90e\") " Nov 03 21:19:01.741393 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.741229 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-workload-socket" pod "c39084a3-cdf2-47b8-8aa1-b09415e6a90e" (UID: "c39084a3-cdf2-47b8-8aa1-b09415e6a90e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:01.741393 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.741291 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-workload-certs" pod "c39084a3-cdf2-47b8-8aa1-b09415e6a90e" (UID: "c39084a3-cdf2-47b8-8aa1-b09415e6a90e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:01.741393 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.741376 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-credential-socket" pod "c39084a3-cdf2-47b8-8aa1-b09415e6a90e" (UID: "c39084a3-cdf2-47b8-8aa1-b09415e6a90e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:01.741564 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.741456 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istiod-ca-cert" pod "c39084a3-cdf2-47b8-8aa1-b09415e6a90e" (UID: "c39084a3-cdf2-47b8-8aa1-b09415e6a90e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:19:01.742666 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.742637 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:01.742795 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.742669 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:01.742795 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.742683 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:01.742795 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.742698 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:01.742795 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.742711 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:01.743762 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.743726 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istio-envoy" pod "c39084a3-cdf2-47b8-8aa1-b09415e6a90e" (UID: "c39084a3-cdf2-47b8-8aa1-b09415e6a90e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:01.743849 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.743760 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istio-podinfo" pod "c39084a3-cdf2-47b8-8aa1-b09415e6a90e" (UID: "c39084a3-cdf2-47b8-8aa1-b09415e6a90e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:19:01.752679 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.752207 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-b7cdc65f9-vsbx8"] Nov 03 21:19:01.752679 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.752460 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-kube-api-access-v9dwq" pod "c39084a3-cdf2-47b8-8aa1-b09415e6a90e" (UID: "c39084a3-cdf2-47b8-8aa1-b09415e6a90e"). InnerVolumeSpecName "kube-api-access-v9dwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:01.752679 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.752506 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istio-token" pod "c39084a3-cdf2-47b8-8aa1-b09415e6a90e" (UID: "c39084a3-cdf2-47b8-8aa1-b09415e6a90e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:01.844182 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.844147 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:01.844182 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.844177 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:01.844182 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.844188 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9dwq\" (UniqueName: \"kubernetes.io/projected/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-kube-api-access-v9dwq\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:01.844428 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:01.844199 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c39084a3-cdf2-47b8-8aa1-b09415e6a90e-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:02.259131 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:02.255855 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np"] Nov 03 21:19:02.259131 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:02.255925 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-32f6e43c-openshift-default-7d9c5fbd5c-bd5np"] Nov 03 21:19:02.549251 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:02.549173 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr" Nov 03 21:19:02.549251 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:02.549246 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh" Nov 03 21:19:03.652640 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:03.652605 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c39084a3-cdf2-47b8-8aa1-b09415e6a90e" path="/var/lib/kubelet/pods/c39084a3-cdf2-47b8-8aa1-b09415e6a90e/volumes" Nov 03 21:19:05.776532 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:05.776495 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr"] Nov 03 21:19:05.776986 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:05.776696 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" podUID="81a14fb4-c56f-4704-91c9-bd2de3648f55" containerName="istio-proxy" containerID="cri-o://dc0acac9d72efcc6f2276daf9541b572ca901ca5699a98d6353c98c890a41a22" gracePeriod=30 Nov 03 21:19:05.824541 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:05.824493 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw"] Nov 03 21:19:05.824799 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:05.824768 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" podUID="ebbedd3d-03f0-4518-b13b-fd88e4c633d5" containerName="istio-proxy" containerID="cri-o://457218192f6b6671c149e4065085b665ee543326e0909e3a0990fa832e93ebee" gracePeriod=30 Nov 03 21:19:05.827133 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:05.827103 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2"] Nov 03 21:19:05.827303 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:05.827282 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" podUID="e3271312-1e59-4a84-8ac1-d8e57f69f85b" containerName="istio-proxy" containerID="cri-o://466fda4383bb15f4883e6330f8b8a5b9a4b1f043bf2cc2210ac5744fdf2519ee" gracePeriod=30 Nov 03 21:19:06.636084 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:06.636028 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-79d96df5c6-cfzcv" podUID="81109a63-4170-413c-bad7-4b49eaf27f0e" containerName="payload-processing" containerID="cri-o://cbe3bbd50f786b7219a9e02dd984fca77dc829f678ad0aecf495a59935a67987" gracePeriod=25 Nov 03 21:19:06.753178 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:06.753122 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-b7cdc65f9-vsbx8" podUID="0d82588a-f1d2-424e-b718-081993dc868c" containerName="payload-pre-processing" containerID="cri-o://946698b96c6c8b5500066ca66c69663a50a0c7f8fbf10b2589f041e70e310267" gracePeriod=25 Nov 03 21:19:06.899090 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:06.899001 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-79d96df5c6-cfzcv" Nov 03 21:19:06.937691 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:06.937381 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81109a63-4170-413c-bad7-4b49eaf27f0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81109a63-4170-413c-bad7-4b49eaf27f0e-plugins-config-volume\") pod \"81109a63-4170-413c-bad7-4b49eaf27f0e\" (UID: \"81109a63-4170-413c-bad7-4b49eaf27f0e\") " Nov 03 21:19:06.937691 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:06.937524 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81109a63-4170-413c-bad7-4b49eaf27f0e-kube-api-access-cm4nh\" (UniqueName: \"kubernetes.io/projected/81109a63-4170-413c-bad7-4b49eaf27f0e-kube-api-access-cm4nh\") pod \"81109a63-4170-413c-bad7-4b49eaf27f0e\" (UID: \"81109a63-4170-413c-bad7-4b49eaf27f0e\") " Nov 03 21:19:06.939813 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:06.939768 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81109a63-4170-413c-bad7-4b49eaf27f0e-plugins-config-volume" pod "81109a63-4170-413c-bad7-4b49eaf27f0e" (UID: "81109a63-4170-413c-bad7-4b49eaf27f0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:19:06.941573 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:06.941545 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81109a63-4170-413c-bad7-4b49eaf27f0e-kube-api-access-cm4nh" pod "81109a63-4170-413c-bad7-4b49eaf27f0e" (UID: "81109a63-4170-413c-bad7-4b49eaf27f0e"). InnerVolumeSpecName "kube-api-access-cm4nh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:07.002467 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:07.002438 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-b7cdc65f9-vsbx8" Nov 03 21:19:07.039758 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:07.039719 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d82588a-f1d2-424e-b718-081993dc868c-kube-api-access-m74h5\" (UniqueName: \"kubernetes.io/projected/0d82588a-f1d2-424e-b718-081993dc868c-kube-api-access-m74h5\") pod \"0d82588a-f1d2-424e-b718-081993dc868c\" (UID: \"0d82588a-f1d2-424e-b718-081993dc868c\") " Nov 03 21:19:07.039930 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:07.039776 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d82588a-f1d2-424e-b718-081993dc868c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d82588a-f1d2-424e-b718-081993dc868c-plugins-config-volume\") pod \"0d82588a-f1d2-424e-b718-081993dc868c\" (UID: \"0d82588a-f1d2-424e-b718-081993dc868c\") " Nov 03 21:19:07.040261 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:07.040234 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d82588a-f1d2-424e-b718-081993dc868c-plugins-config-volume" pod "0d82588a-f1d2-424e-b718-081993dc868c" (UID: "0d82588a-f1d2-424e-b718-081993dc868c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:19:07.040404 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:07.040386 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81109a63-4170-413c-bad7-4b49eaf27f0e-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:07.040465 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:07.040406 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm4nh\" (UniqueName: \"kubernetes.io/projected/81109a63-4170-413c-bad7-4b49eaf27f0e-kube-api-access-cm4nh\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:07.040465 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:07.040416 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d82588a-f1d2-424e-b718-081993dc868c-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:07.042157 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:07.042127 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d82588a-f1d2-424e-b718-081993dc868c-kube-api-access-m74h5" pod "0d82588a-f1d2-424e-b718-081993dc868c" (UID: "0d82588a-f1d2-424e-b718-081993dc868c"). InnerVolumeSpecName "kube-api-access-m74h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:07.141771 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:07.141733 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m74h5\" (UniqueName: \"kubernetes.io/projected/0d82588a-f1d2-424e-b718-081993dc868c-kube-api-access-m74h5\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:07.236157 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:07.236120 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd"] Nov 03 21:19:07.236355 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:07.236334 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" podUID="11b3bf8b-24eb-4b93-80a1-fabdf14cff21" containerName="istio-proxy" containerID="cri-o://6a95a3fe50fd2cf9bd879ce6e5cbb94c7cdce5cac048311030c45c169c135277" gracePeriod=30 Nov 03 21:19:07.683244 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:07.683209 5615 generic.go:356] "Generic (PLEG): container finished" podID="0d82588a-f1d2-424e-b718-081993dc868c" containerID="946698b96c6c8b5500066ca66c69663a50a0c7f8fbf10b2589f041e70e310267" exitCode=0 Nov 03 21:19:07.683397 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:07.683253 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-b7cdc65f9-vsbx8" Nov 03 21:19:07.683397 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:07.683291 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-b7cdc65f9-vsbx8" event={"ID":"0d82588a-f1d2-424e-b718-081993dc868c","Type":"ContainerDied","Data":"946698b96c6c8b5500066ca66c69663a50a0c7f8fbf10b2589f041e70e310267"} Nov 03 21:19:07.683397 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:07.683345 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-b7cdc65f9-vsbx8" event={"ID":"0d82588a-f1d2-424e-b718-081993dc868c","Type":"ContainerDied","Data":"d679b64ada038b02069c0db37e73fd08fdffc9db0aa70788c3fe9b68e0db7fc9"} Nov 03 21:19:07.683397 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:07.683369 5615 scope.go:122] "RemoveContainer" containerID="946698b96c6c8b5500066ca66c69663a50a0c7f8fbf10b2589f041e70e310267" Nov 03 21:19:07.684770 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:07.684747 5615 generic.go:356] "Generic (PLEG): container finished" podID="81109a63-4170-413c-bad7-4b49eaf27f0e" containerID="cbe3bbd50f786b7219a9e02dd984fca77dc829f678ad0aecf495a59935a67987" exitCode=0 Nov 03 21:19:07.684879 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:07.684792 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-79d96df5c6-cfzcv" Nov 03 21:19:07.684923 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:07.684781 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-79d96df5c6-cfzcv" event={"ID":"81109a63-4170-413c-bad7-4b49eaf27f0e","Type":"ContainerDied","Data":"cbe3bbd50f786b7219a9e02dd984fca77dc829f678ad0aecf495a59935a67987"} Nov 03 21:19:07.684923 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:07.684916 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-79d96df5c6-cfzcv" event={"ID":"81109a63-4170-413c-bad7-4b49eaf27f0e","Type":"ContainerDied","Data":"44b2488f45b5bfd57abd37243d3106cb4394cf54be3444811b5634d2b1f4b41f"} Nov 03 21:19:07.693652 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:07.693629 5615 scope.go:122] "RemoveContainer" containerID="946698b96c6c8b5500066ca66c69663a50a0c7f8fbf10b2589f041e70e310267" Nov 03 21:19:07.693932 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:19:07.693913 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"946698b96c6c8b5500066ca66c69663a50a0c7f8fbf10b2589f041e70e310267\": container with ID starting with 946698b96c6c8b5500066ca66c69663a50a0c7f8fbf10b2589f041e70e310267 not found: ID does not exist" containerID="946698b96c6c8b5500066ca66c69663a50a0c7f8fbf10b2589f041e70e310267" Nov 03 21:19:07.693981 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:07.693941 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"946698b96c6c8b5500066ca66c69663a50a0c7f8fbf10b2589f041e70e310267"} err="failed to get container status \"946698b96c6c8b5500066ca66c69663a50a0c7f8fbf10b2589f041e70e310267\": rpc error: code = NotFound desc = could not find container \"946698b96c6c8b5500066ca66c69663a50a0c7f8fbf10b2589f041e70e310267\": container with ID starting with 946698b96c6c8b5500066ca66c69663a50a0c7f8fbf10b2589f041e70e310267 not found: ID does not exist" Nov 03 21:19:07.693981 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:07.693957 5615 scope.go:122] "RemoveContainer" containerID="cbe3bbd50f786b7219a9e02dd984fca77dc829f678ad0aecf495a59935a67987" Nov 03 21:19:07.703010 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:07.702993 5615 scope.go:122] "RemoveContainer" containerID="cbe3bbd50f786b7219a9e02dd984fca77dc829f678ad0aecf495a59935a67987" Nov 03 21:19:07.703300 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:19:07.703279 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbe3bbd50f786b7219a9e02dd984fca77dc829f678ad0aecf495a59935a67987\": container with ID starting with cbe3bbd50f786b7219a9e02dd984fca77dc829f678ad0aecf495a59935a67987 not found: ID does not exist" containerID="cbe3bbd50f786b7219a9e02dd984fca77dc829f678ad0aecf495a59935a67987" Nov 03 21:19:07.703379 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:07.703309 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbe3bbd50f786b7219a9e02dd984fca77dc829f678ad0aecf495a59935a67987"} err="failed to get container status \"cbe3bbd50f786b7219a9e02dd984fca77dc829f678ad0aecf495a59935a67987\": rpc error: code = NotFound desc = could not find container \"cbe3bbd50f786b7219a9e02dd984fca77dc829f678ad0aecf495a59935a67987\": container with ID starting with cbe3bbd50f786b7219a9e02dd984fca77dc829f678ad0aecf495a59935a67987 not found: ID does not exist" Nov 03 21:19:07.847345 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:07.847307 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-b7cdc65f9-vsbx8"] Nov 03 21:19:07.850953 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:07.850919 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-b7cdc65f9-vsbx8"] Nov 03 21:19:07.954405 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:07.954319 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-79d96df5c6-cfzcv"] Nov 03 21:19:07.958471 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:07.958433 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-79d96df5c6-cfzcv"] Nov 03 21:19:08.091948 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:08.091909 5615 patch_prober.go:29] interesting pod/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:19:08.092145 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:08.091969 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" podUID="11b3bf8b-24eb-4b93-80a1-fabdf14cff21" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:19:08.759440 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:08.759403 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:19:08.759637 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:08.759473 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" podUID="ebbedd3d-03f0-4518-b13b-fd88e4c633d5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:19:09.022933 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:09.022856 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:19:09.022933 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:09.022912 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" podUID="81a14fb4-c56f-4704-91c9-bd2de3648f55" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:19:09.176649 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:09.176608 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-msdt2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:19:09.176844 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:09.176673 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" podUID="e3271312-1e59-4a84-8ac1-d8e57f69f85b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:19:09.652013 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:09.651980 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d82588a-f1d2-424e-b718-081993dc868c" path="/var/lib/kubelet/pods/0d82588a-f1d2-424e-b718-081993dc868c/volumes" Nov 03 21:19:09.652484 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:09.652468 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81109a63-4170-413c-bad7-4b49eaf27f0e" path="/var/lib/kubelet/pods/81109a63-4170-413c-bad7-4b49eaf27f0e/volumes" Nov 03 21:19:10.619360 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:19:10.619324 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:19:11.119362 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.119325 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:19:11.161285 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.161254 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:19:11.164823 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.164802 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:19:11.192515 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.192442 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istiod-ca-cert\") pod \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " Nov 03 21:19:11.192515 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.192486 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istio-token\" (UniqueName: \"kubernetes.io/projected/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istio-token\") pod \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " Nov 03 21:19:11.192515 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.192513 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a14fb4-c56f-4704-91c9-bd2de3648f55-istio-token\" (UniqueName: \"kubernetes.io/projected/81a14fb4-c56f-4704-91c9-bd2de3648f55-istio-token\") pod \"81a14fb4-c56f-4704-91c9-bd2de3648f55\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " Nov 03 21:19:11.192772 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.192533 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-workload-certs\") pod \"81a14fb4-c56f-4704-91c9-bd2de3648f55\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " Nov 03 21:19:11.192772 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.192567 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-credential-socket\") pod \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " Nov 03 21:19:11.192772 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.192598 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-workload-socket\") pod \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " Nov 03 21:19:11.192772 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.192623 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-kube-api-access-64lp2\" (UniqueName: \"kubernetes.io/projected/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-kube-api-access-64lp2\") pod \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " Nov 03 21:19:11.192772 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.192663 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-credential-socket\") pod \"81a14fb4-c56f-4704-91c9-bd2de3648f55\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " Nov 03 21:19:11.192772 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.192707 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istiod-ca-cert\") pod \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " Nov 03 21:19:11.192772 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.192746 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a14fb4-c56f-4704-91c9-bd2de3648f55-kube-api-access-wgt9n\" (UniqueName: \"kubernetes.io/projected/81a14fb4-c56f-4704-91c9-bd2de3648f55-kube-api-access-wgt9n\") pod \"81a14fb4-c56f-4704-91c9-bd2de3648f55\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " Nov 03 21:19:11.193158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.192784 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istio-podinfo\") pod \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " Nov 03 21:19:11.193158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.192812 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-workload-certs" pod "81a14fb4-c56f-4704-91c9-bd2de3648f55" (UID: "81a14fb4-c56f-4704-91c9-bd2de3648f55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:11.193158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.192835 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-workload-certs\") pod \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " Nov 03 21:19:11.193158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.192872 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istio-data\") pod \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " Nov 03 21:19:11.193158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.192916 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-credential-socket\") pod \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " Nov 03 21:19:11.193158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.192947 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81a14fb4-c56f-4704-91c9-bd2de3648f55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a14fb4-c56f-4704-91c9-bd2de3648f55-istio-podinfo\") pod \"81a14fb4-c56f-4704-91c9-bd2de3648f55\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " Nov 03 21:19:11.193158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.192985 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-workload-socket\") pod \"81a14fb4-c56f-4704-91c9-bd2de3648f55\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " Nov 03 21:19:11.193158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.193059 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-istio-envoy\") pod \"81a14fb4-c56f-4704-91c9-bd2de3648f55\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " Nov 03 21:19:11.193158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.193101 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3271312-1e59-4a84-8ac1-d8e57f69f85b-kube-api-access-zbmzs\" (UniqueName: \"kubernetes.io/projected/e3271312-1e59-4a84-8ac1-d8e57f69f85b-kube-api-access-zbmzs\") pod \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " Nov 03 21:19:11.193158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.193143 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81a14fb4-c56f-4704-91c9-bd2de3648f55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a14fb4-c56f-4704-91c9-bd2de3648f55-istiod-ca-cert\") pod \"81a14fb4-c56f-4704-91c9-bd2de3648f55\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " Nov 03 21:19:11.193662 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.193159 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-workload-socket" pod "ebbedd3d-03f0-4518-b13b-fd88e4c633d5" (UID: "ebbedd3d-03f0-4518-b13b-fd88e4c633d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:11.193662 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.193190 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istio-envoy\") pod \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " Nov 03 21:19:11.193662 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.193220 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-istio-data\") pod \"81a14fb4-c56f-4704-91c9-bd2de3648f55\" (UID: \"81a14fb4-c56f-4704-91c9-bd2de3648f55\") " Nov 03 21:19:11.193662 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.193259 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istio-token\" (UniqueName: \"kubernetes.io/projected/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istio-token\") pod \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " Nov 03 21:19:11.193662 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.193257 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-credential-socket" pod "81a14fb4-c56f-4704-91c9-bd2de3648f55" (UID: "81a14fb4-c56f-4704-91c9-bd2de3648f55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:11.193662 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.193308 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istio-data\") pod \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " Nov 03 21:19:11.193662 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.193354 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-workload-certs\") pod \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " Nov 03 21:19:11.193662 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.193397 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-workload-socket" pod "81a14fb4-c56f-4704-91c9-bd2de3648f55" (UID: "81a14fb4-c56f-4704-91c9-bd2de3648f55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:11.193662 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.193423 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-workload-socket\") pod \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " Nov 03 21:19:11.193662 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.193459 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-credential-socket" pod "ebbedd3d-03f0-4518-b13b-fd88e4c633d5" (UID: "ebbedd3d-03f0-4518-b13b-fd88e4c633d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:11.193662 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.193478 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istio-envoy\") pod \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\" (UID: \"e3271312-1e59-4a84-8ac1-d8e57f69f85b\") " Nov 03 21:19:11.193662 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.193517 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istio-podinfo\") pod \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\" (UID: \"ebbedd3d-03f0-4518-b13b-fd88e4c633d5\") " Nov 03 21:19:11.193662 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.193583 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istio-data" pod "e3271312-1e59-4a84-8ac1-d8e57f69f85b" (UID: "e3271312-1e59-4a84-8ac1-d8e57f69f85b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:11.193662 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.193610 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-credential-socket" pod "e3271312-1e59-4a84-8ac1-d8e57f69f85b" (UID: "e3271312-1e59-4a84-8ac1-d8e57f69f85b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:11.194446 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.193691 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-istio-data" pod "81a14fb4-c56f-4704-91c9-bd2de3648f55" (UID: "81a14fb4-c56f-4704-91c9-bd2de3648f55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:11.194446 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.193762 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-workload-certs" pod "e3271312-1e59-4a84-8ac1-d8e57f69f85b" (UID: "e3271312-1e59-4a84-8ac1-d8e57f69f85b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:11.194446 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.193923 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istiod-ca-cert" pod "ebbedd3d-03f0-4518-b13b-fd88e4c633d5" (UID: "ebbedd3d-03f0-4518-b13b-fd88e4c633d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:19:11.194446 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.194288 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:11.194446 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.194312 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:11.194446 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.194328 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:11.194446 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.194342 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:11.194446 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.194356 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:11.194446 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.194370 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:11.194446 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.194385 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:11.194446 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.194398 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:11.194446 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.194414 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:11.194446 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.194324 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81a14fb4-c56f-4704-91c9-bd2de3648f55-istiod-ca-cert" pod "81a14fb4-c56f-4704-91c9-bd2de3648f55" (UID: "81a14fb4-c56f-4704-91c9-bd2de3648f55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:19:11.195152 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.194648 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-workload-socket" pod "e3271312-1e59-4a84-8ac1-d8e57f69f85b" (UID: "e3271312-1e59-4a84-8ac1-d8e57f69f85b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:11.195356 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.195324 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istio-data" pod "ebbedd3d-03f0-4518-b13b-fd88e4c633d5" (UID: "ebbedd3d-03f0-4518-b13b-fd88e4c633d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:11.195434 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.195367 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-workload-certs" pod "ebbedd3d-03f0-4518-b13b-fd88e4c633d5" (UID: "ebbedd3d-03f0-4518-b13b-fd88e4c633d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:11.195809 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.195785 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istiod-ca-cert" pod "e3271312-1e59-4a84-8ac1-d8e57f69f85b" (UID: "e3271312-1e59-4a84-8ac1-d8e57f69f85b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:19:11.195936 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.195913 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a14fb4-c56f-4704-91c9-bd2de3648f55-istio-token" pod "81a14fb4-c56f-4704-91c9-bd2de3648f55" (UID: "81a14fb4-c56f-4704-91c9-bd2de3648f55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:11.196463 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.196437 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istio-token" pod "e3271312-1e59-4a84-8ac1-d8e57f69f85b" (UID: "e3271312-1e59-4a84-8ac1-d8e57f69f85b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:11.196619 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.196590 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a14fb4-c56f-4704-91c9-bd2de3648f55-kube-api-access-wgt9n" pod "81a14fb4-c56f-4704-91c9-bd2de3648f55" (UID: "81a14fb4-c56f-4704-91c9-bd2de3648f55"). InnerVolumeSpecName "kube-api-access-wgt9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:11.197162 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.197136 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-istio-envoy" pod "81a14fb4-c56f-4704-91c9-bd2de3648f55" (UID: "81a14fb4-c56f-4704-91c9-bd2de3648f55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:11.197162 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.197152 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81a14fb4-c56f-4704-91c9-bd2de3648f55-istio-podinfo" pod "81a14fb4-c56f-4704-91c9-bd2de3648f55" (UID: "81a14fb4-c56f-4704-91c9-bd2de3648f55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:19:11.197623 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.197598 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istio-podinfo" pod "ebbedd3d-03f0-4518-b13b-fd88e4c633d5" (UID: "ebbedd3d-03f0-4518-b13b-fd88e4c633d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:19:11.197714 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.197634 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3271312-1e59-4a84-8ac1-d8e57f69f85b-kube-api-access-zbmzs" pod "e3271312-1e59-4a84-8ac1-d8e57f69f85b" (UID: "e3271312-1e59-4a84-8ac1-d8e57f69f85b"). InnerVolumeSpecName "kube-api-access-zbmzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:11.197936 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.197910 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istio-envoy" pod "ebbedd3d-03f0-4518-b13b-fd88e4c633d5" (UID: "ebbedd3d-03f0-4518-b13b-fd88e4c633d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:11.198031 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.197986 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istio-envoy" pod "e3271312-1e59-4a84-8ac1-d8e57f69f85b" (UID: "e3271312-1e59-4a84-8ac1-d8e57f69f85b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:11.198031 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.198009 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istio-token" pod "ebbedd3d-03f0-4518-b13b-fd88e4c633d5" (UID: "ebbedd3d-03f0-4518-b13b-fd88e4c633d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:11.198382 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.198367 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-kube-api-access-64lp2" pod "ebbedd3d-03f0-4518-b13b-fd88e4c633d5" (UID: "ebbedd3d-03f0-4518-b13b-fd88e4c633d5"). InnerVolumeSpecName "kube-api-access-64lp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:11.198448 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.198433 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istio-podinfo" pod "e3271312-1e59-4a84-8ac1-d8e57f69f85b" (UID: "e3271312-1e59-4a84-8ac1-d8e57f69f85b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:19:11.295456 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.295420 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:11.295456 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.295453 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbmzs\" (UniqueName: \"kubernetes.io/projected/e3271312-1e59-4a84-8ac1-d8e57f69f85b-kube-api-access-zbmzs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:11.295635 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.295467 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a14fb4-c56f-4704-91c9-bd2de3648f55-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:11.295635 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.295480 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:11.295635 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.295490 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:11.295635 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.295500 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:11.295635 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.295509 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:11.295635 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.295518 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:11.295635 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.295526 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:11.295635 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.295535 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:11.295635 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.295548 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:11.295635 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.295559 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:11.295635 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.295567 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a14fb4-c56f-4704-91c9-bd2de3648f55-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:11.295635 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.295577 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64lp2\" (UniqueName: \"kubernetes.io/projected/ebbedd3d-03f0-4518-b13b-fd88e4c633d5-kube-api-access-64lp2\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:11.295635 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.295586 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgt9n\" (UniqueName: \"kubernetes.io/projected/81a14fb4-c56f-4704-91c9-bd2de3648f55-kube-api-access-wgt9n\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:11.295635 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.295596 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3271312-1e59-4a84-8ac1-d8e57f69f85b-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:11.295635 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.295606 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a14fb4-c56f-4704-91c9-bd2de3648f55-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:11.295635 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.295616 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a14fb4-c56f-4704-91c9-bd2de3648f55-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:11.716030 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.715993 5615 generic.go:356] "Generic (PLEG): container finished" podID="e3271312-1e59-4a84-8ac1-d8e57f69f85b" containerID="466fda4383bb15f4883e6330f8b8a5b9a4b1f043bf2cc2210ac5744fdf2519ee" exitCode=0 Nov 03 21:19:11.716482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.716045 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" Nov 03 21:19:11.716482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.716088 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" event={"ID":"e3271312-1e59-4a84-8ac1-d8e57f69f85b","Type":"ContainerDied","Data":"466fda4383bb15f4883e6330f8b8a5b9a4b1f043bf2cc2210ac5744fdf2519ee"} Nov 03 21:19:11.716482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.716151 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2" event={"ID":"e3271312-1e59-4a84-8ac1-d8e57f69f85b","Type":"ContainerDied","Data":"b1bdc40c6ff8a1affa342546e0b61ee919282c6d3a0190f0ffe36fa5aa360311"} Nov 03 21:19:11.716482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.716179 5615 scope.go:122] "RemoveContainer" containerID="466fda4383bb15f4883e6330f8b8a5b9a4b1f043bf2cc2210ac5744fdf2519ee" Nov 03 21:19:11.717812 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.717785 5615 generic.go:356] "Generic (PLEG): container finished" podID="81a14fb4-c56f-4704-91c9-bd2de3648f55" containerID="dc0acac9d72efcc6f2276daf9541b572ca901ca5699a98d6353c98c890a41a22" exitCode=0 Nov 03 21:19:11.717931 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.717831 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" Nov 03 21:19:11.717931 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.717873 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" event={"ID":"81a14fb4-c56f-4704-91c9-bd2de3648f55","Type":"ContainerDied","Data":"dc0acac9d72efcc6f2276daf9541b572ca901ca5699a98d6353c98c890a41a22"} Nov 03 21:19:11.718039 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.717929 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr" event={"ID":"81a14fb4-c56f-4704-91c9-bd2de3648f55","Type":"ContainerDied","Data":"ca075319251623d38a89665469f86a3731c874143e3a8a6a8b8f7556cdbfc095"} Nov 03 21:19:11.719848 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.719828 5615 generic.go:356] "Generic (PLEG): container finished" podID="ebbedd3d-03f0-4518-b13b-fd88e4c633d5" containerID="457218192f6b6671c149e4065085b665ee543326e0909e3a0990fa832e93ebee" exitCode=0 Nov 03 21:19:11.719946 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.719862 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" event={"ID":"ebbedd3d-03f0-4518-b13b-fd88e4c633d5","Type":"ContainerDied","Data":"457218192f6b6671c149e4065085b665ee543326e0909e3a0990fa832e93ebee"} Nov 03 21:19:11.719946 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.719887 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" Nov 03 21:19:11.720051 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.719894 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw" event={"ID":"ebbedd3d-03f0-4518-b13b-fd88e4c633d5","Type":"ContainerDied","Data":"e24bb392976f91973b5f0b86a0cc071587c55c89f5cab18c17e803ce3adc0c10"} Nov 03 21:19:11.727282 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.727256 5615 scope.go:122] "RemoveContainer" containerID="466fda4383bb15f4883e6330f8b8a5b9a4b1f043bf2cc2210ac5744fdf2519ee" Nov 03 21:19:11.727589 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:19:11.727571 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"466fda4383bb15f4883e6330f8b8a5b9a4b1f043bf2cc2210ac5744fdf2519ee\": container with ID starting with 466fda4383bb15f4883e6330f8b8a5b9a4b1f043bf2cc2210ac5744fdf2519ee not found: ID does not exist" containerID="466fda4383bb15f4883e6330f8b8a5b9a4b1f043bf2cc2210ac5744fdf2519ee" Nov 03 21:19:11.727647 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.727598 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"466fda4383bb15f4883e6330f8b8a5b9a4b1f043bf2cc2210ac5744fdf2519ee"} err="failed to get container status \"466fda4383bb15f4883e6330f8b8a5b9a4b1f043bf2cc2210ac5744fdf2519ee\": rpc error: code = NotFound desc = could not find container \"466fda4383bb15f4883e6330f8b8a5b9a4b1f043bf2cc2210ac5744fdf2519ee\": container with ID starting with 466fda4383bb15f4883e6330f8b8a5b9a4b1f043bf2cc2210ac5744fdf2519ee not found: ID does not exist" Nov 03 21:19:11.727647 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.727615 5615 scope.go:122] "RemoveContainer" containerID="dc0acac9d72efcc6f2276daf9541b572ca901ca5699a98d6353c98c890a41a22" Nov 03 21:19:11.736679 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.736661 5615 scope.go:122] "RemoveContainer" containerID="dc0acac9d72efcc6f2276daf9541b572ca901ca5699a98d6353c98c890a41a22" Nov 03 21:19:11.736970 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:19:11.736947 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc0acac9d72efcc6f2276daf9541b572ca901ca5699a98d6353c98c890a41a22\": container with ID starting with dc0acac9d72efcc6f2276daf9541b572ca901ca5699a98d6353c98c890a41a22 not found: ID does not exist" containerID="dc0acac9d72efcc6f2276daf9541b572ca901ca5699a98d6353c98c890a41a22" Nov 03 21:19:11.737053 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.736976 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc0acac9d72efcc6f2276daf9541b572ca901ca5699a98d6353c98c890a41a22"} err="failed to get container status \"dc0acac9d72efcc6f2276daf9541b572ca901ca5699a98d6353c98c890a41a22\": rpc error: code = NotFound desc = could not find container \"dc0acac9d72efcc6f2276daf9541b572ca901ca5699a98d6353c98c890a41a22\": container with ID starting with dc0acac9d72efcc6f2276daf9541b572ca901ca5699a98d6353c98c890a41a22 not found: ID does not exist" Nov 03 21:19:11.737053 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.736999 5615 scope.go:122] "RemoveContainer" containerID="457218192f6b6671c149e4065085b665ee543326e0909e3a0990fa832e93ebee" Nov 03 21:19:11.745834 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.745815 5615 scope.go:122] "RemoveContainer" containerID="457218192f6b6671c149e4065085b665ee543326e0909e3a0990fa832e93ebee" Nov 03 21:19:11.746141 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:19:11.746120 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"457218192f6b6671c149e4065085b665ee543326e0909e3a0990fa832e93ebee\": container with ID starting with 457218192f6b6671c149e4065085b665ee543326e0909e3a0990fa832e93ebee not found: ID does not exist" containerID="457218192f6b6671c149e4065085b665ee543326e0909e3a0990fa832e93ebee" Nov 03 21:19:11.746195 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.746148 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457218192f6b6671c149e4065085b665ee543326e0909e3a0990fa832e93ebee"} err="failed to get container status \"457218192f6b6671c149e4065085b665ee543326e0909e3a0990fa832e93ebee\": rpc error: code = NotFound desc = could not find container \"457218192f6b6671c149e4065085b665ee543326e0909e3a0990fa832e93ebee\": container with ID starting with 457218192f6b6671c149e4065085b665ee543326e0909e3a0990fa832e93ebee not found: ID does not exist" Nov 03 21:19:11.875432 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.875395 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2"] Nov 03 21:19:11.878591 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:11.878556 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-msdt2"] Nov 03 21:19:12.001243 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.001160 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw"] Nov 03 21:19:12.003191 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.003160 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-l9qgw"] Nov 03 21:19:12.093683 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.093644 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr"] Nov 03 21:19:12.099156 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.099116 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hzgjr"] Nov 03 21:19:12.507334 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.507306 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:19:12.619484 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.619399 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-kube-api-access-662h7\" (UniqueName: \"kubernetes.io/projected/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-kube-api-access-662h7\") pod \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " Nov 03 21:19:12.619484 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.619445 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-credential-socket\") pod \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " Nov 03 21:19:12.619655 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.619537 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istio-data\") pod \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " Nov 03 21:19:12.619655 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.619631 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-workload-socket\") pod \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " Nov 03 21:19:12.619736 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.619662 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istio-token\" (UniqueName: \"kubernetes.io/projected/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istio-token\") pod \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " Nov 03 21:19:12.619777 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.619732 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istiod-ca-cert\") pod \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " Nov 03 21:19:12.619814 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.619786 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istio-envoy\") pod \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " Nov 03 21:19:12.619863 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.619840 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istio-podinfo\") pod \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " Nov 03 21:19:12.619863 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.619847 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istio-data" pod "11b3bf8b-24eb-4b93-80a1-fabdf14cff21" (UID: "11b3bf8b-24eb-4b93-80a1-fabdf14cff21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:12.619975 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.619844 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-credential-socket" pod "11b3bf8b-24eb-4b93-80a1-fabdf14cff21" (UID: "11b3bf8b-24eb-4b93-80a1-fabdf14cff21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:12.619975 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.619893 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-workload-socket" pod "11b3bf8b-24eb-4b93-80a1-fabdf14cff21" (UID: "11b3bf8b-24eb-4b93-80a1-fabdf14cff21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:12.619975 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.619901 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-workload-certs\") pod \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\" (UID: \"11b3bf8b-24eb-4b93-80a1-fabdf14cff21\") " Nov 03 21:19:12.620221 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.620196 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istiod-ca-cert" pod "11b3bf8b-24eb-4b93-80a1-fabdf14cff21" (UID: "11b3bf8b-24eb-4b93-80a1-fabdf14cff21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:19:12.620371 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.620345 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-workload-certs" pod "11b3bf8b-24eb-4b93-80a1-fabdf14cff21" (UID: "11b3bf8b-24eb-4b93-80a1-fabdf14cff21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:12.620659 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.620632 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:12.620747 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.620661 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:12.620747 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.620676 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:12.620747 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.620690 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:12.620747 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.620704 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:12.621883 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.621853 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istio-envoy" pod "11b3bf8b-24eb-4b93-80a1-fabdf14cff21" (UID: "11b3bf8b-24eb-4b93-80a1-fabdf14cff21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:12.621954 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.621874 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-kube-api-access-662h7" pod "11b3bf8b-24eb-4b93-80a1-fabdf14cff21" (UID: "11b3bf8b-24eb-4b93-80a1-fabdf14cff21"). InnerVolumeSpecName "kube-api-access-662h7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:12.621954 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.621890 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istio-token" pod "11b3bf8b-24eb-4b93-80a1-fabdf14cff21" (UID: "11b3bf8b-24eb-4b93-80a1-fabdf14cff21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:12.622123 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.622051 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istio-podinfo" pod "11b3bf8b-24eb-4b93-80a1-fabdf14cff21" (UID: "11b3bf8b-24eb-4b93-80a1-fabdf14cff21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:19:12.722306 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.722269 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-662h7\" (UniqueName: \"kubernetes.io/projected/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-kube-api-access-662h7\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:12.722855 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.722550 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:12.722855 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.722578 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:12.722855 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.722621 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b3bf8b-24eb-4b93-80a1-fabdf14cff21-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:12.731318 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.731285 5615 generic.go:356] "Generic (PLEG): container finished" podID="11b3bf8b-24eb-4b93-80a1-fabdf14cff21" containerID="6a95a3fe50fd2cf9bd879ce6e5cbb94c7cdce5cac048311030c45c169c135277" exitCode=0 Nov 03 21:19:12.731456 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.731338 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" Nov 03 21:19:12.731456 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.731352 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" event={"ID":"11b3bf8b-24eb-4b93-80a1-fabdf14cff21","Type":"ContainerDied","Data":"6a95a3fe50fd2cf9bd879ce6e5cbb94c7cdce5cac048311030c45c169c135277"} Nov 03 21:19:12.731456 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.731392 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd" event={"ID":"11b3bf8b-24eb-4b93-80a1-fabdf14cff21","Type":"ContainerDied","Data":"142d09c80e8017ef51249ae1a18b613f7a6d99a7410b140104253e23a8635e6d"} Nov 03 21:19:12.731456 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.731416 5615 scope.go:122] "RemoveContainer" containerID="6a95a3fe50fd2cf9bd879ce6e5cbb94c7cdce5cac048311030c45c169c135277" Nov 03 21:19:12.743852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.743819 5615 scope.go:122] "RemoveContainer" containerID="5da03baf1f33f94c77a73c9a13a54392fcf186b52f9c17a702b785901e25abd4" Nov 03 21:19:12.753926 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.753899 5615 scope.go:122] "RemoveContainer" containerID="6a95a3fe50fd2cf9bd879ce6e5cbb94c7cdce5cac048311030c45c169c135277" Nov 03 21:19:12.754278 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:19:12.754253 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a95a3fe50fd2cf9bd879ce6e5cbb94c7cdce5cac048311030c45c169c135277\": container with ID starting with 6a95a3fe50fd2cf9bd879ce6e5cbb94c7cdce5cac048311030c45c169c135277 not found: ID does not exist" containerID="6a95a3fe50fd2cf9bd879ce6e5cbb94c7cdce5cac048311030c45c169c135277" Nov 03 21:19:12.754366 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.754288 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a95a3fe50fd2cf9bd879ce6e5cbb94c7cdce5cac048311030c45c169c135277"} err="failed to get container status \"6a95a3fe50fd2cf9bd879ce6e5cbb94c7cdce5cac048311030c45c169c135277\": rpc error: code = NotFound desc = could not find container \"6a95a3fe50fd2cf9bd879ce6e5cbb94c7cdce5cac048311030c45c169c135277\": container with ID starting with 6a95a3fe50fd2cf9bd879ce6e5cbb94c7cdce5cac048311030c45c169c135277 not found: ID does not exist" Nov 03 21:19:12.754366 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.754310 5615 scope.go:122] "RemoveContainer" containerID="5da03baf1f33f94c77a73c9a13a54392fcf186b52f9c17a702b785901e25abd4" Nov 03 21:19:12.754543 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:19:12.754528 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5da03baf1f33f94c77a73c9a13a54392fcf186b52f9c17a702b785901e25abd4\": container with ID starting with 5da03baf1f33f94c77a73c9a13a54392fcf186b52f9c17a702b785901e25abd4 not found: ID does not exist" containerID="5da03baf1f33f94c77a73c9a13a54392fcf186b52f9c17a702b785901e25abd4" Nov 03 21:19:12.754585 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.754549 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5da03baf1f33f94c77a73c9a13a54392fcf186b52f9c17a702b785901e25abd4"} err="failed to get container status \"5da03baf1f33f94c77a73c9a13a54392fcf186b52f9c17a702b785901e25abd4\": rpc error: code = NotFound desc = could not find container \"5da03baf1f33f94c77a73c9a13a54392fcf186b52f9c17a702b785901e25abd4\": container with ID starting with 5da03baf1f33f94c77a73c9a13a54392fcf186b52f9c17a702b785901e25abd4 not found: ID does not exist" Nov 03 21:19:12.900773 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.900637 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd"] Nov 03 21:19:12.902662 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:12.902626 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-hlkqd"] Nov 03 21:19:13.651422 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:13.651389 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11b3bf8b-24eb-4b93-80a1-fabdf14cff21" path="/var/lib/kubelet/pods/11b3bf8b-24eb-4b93-80a1-fabdf14cff21/volumes" Nov 03 21:19:13.651878 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:13.651864 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81a14fb4-c56f-4704-91c9-bd2de3648f55" path="/var/lib/kubelet/pods/81a14fb4-c56f-4704-91c9-bd2de3648f55/volumes" Nov 03 21:19:13.652320 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:13.652307 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3271312-1e59-4a84-8ac1-d8e57f69f85b" path="/var/lib/kubelet/pods/e3271312-1e59-4a84-8ac1-d8e57f69f85b/volumes" Nov 03 21:19:13.653034 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:13.653019 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebbedd3d-03f0-4518-b13b-fd88e4c633d5" path="/var/lib/kubelet/pods/ebbedd3d-03f0-4518-b13b-fd88e4c633d5/volumes" Nov 03 21:19:17.005646 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.005612 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj"] Nov 03 21:19:17.007399 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.007370 5615 cpu_manager.go:414] "containerMap: removing container" podUID="0d82588a-f1d2-424e-b718-081993dc868c" containerName="payload-pre-processing" Nov 03 21:19:17.007575 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.007558 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d82588a-f1d2-424e-b718-081993dc868c" containerName="payload-pre-processing" Nov 03 21:19:17.007703 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.007677 5615 cpu_manager.go:414] "containerMap: removing container" podUID="81109a63-4170-413c-bad7-4b49eaf27f0e" containerName="payload-processing" Nov 03 21:19:17.007703 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.007697 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81109a63-4170-413c-bad7-4b49eaf27f0e" containerName="payload-processing" Nov 03 21:19:17.007902 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.007726 5615 cpu_manager.go:414] "containerMap: removing container" podUID="11b3bf8b-24eb-4b93-80a1-fabdf14cff21" containerName="istio-proxy" Nov 03 21:19:17.007902 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.007739 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11b3bf8b-24eb-4b93-80a1-fabdf14cff21" containerName="istio-proxy" Nov 03 21:19:17.007902 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.007755 5615 cpu_manager.go:414] "containerMap: removing container" podUID="c39084a3-cdf2-47b8-8aa1-b09415e6a90e" containerName="istio-proxy" Nov 03 21:19:17.007902 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.007764 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c39084a3-cdf2-47b8-8aa1-b09415e6a90e" containerName="istio-proxy" Nov 03 21:19:17.007902 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.007812 5615 cpu_manager.go:414] "containerMap: removing container" podUID="ebbedd3d-03f0-4518-b13b-fd88e4c633d5" containerName="istio-proxy" Nov 03 21:19:17.007902 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.007820 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebbedd3d-03f0-4518-b13b-fd88e4c633d5" containerName="istio-proxy" Nov 03 21:19:17.007902 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.007833 5615 cpu_manager.go:414] "containerMap: removing container" podUID="c39084a3-cdf2-47b8-8aa1-b09415e6a90e" containerName="istio-proxy" Nov 03 21:19:17.007902 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.007841 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c39084a3-cdf2-47b8-8aa1-b09415e6a90e" containerName="istio-proxy" Nov 03 21:19:17.007902 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.007858 5615 cpu_manager.go:414] "containerMap: removing container" podUID="11b3bf8b-24eb-4b93-80a1-fabdf14cff21" containerName="istio-proxy" Nov 03 21:19:17.007902 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.007865 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11b3bf8b-24eb-4b93-80a1-fabdf14cff21" containerName="istio-proxy" Nov 03 21:19:17.007902 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.007880 5615 cpu_manager.go:414] "containerMap: removing container" podUID="6c156401-fe0c-4439-b1a8-7c9bdd23a624" containerName="maas-api" Nov 03 21:19:17.007902 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.007887 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c156401-fe0c-4439-b1a8-7c9bdd23a624" containerName="maas-api" Nov 03 21:19:17.007902 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.007901 5615 cpu_manager.go:414] "containerMap: removing container" podUID="81a14fb4-c56f-4704-91c9-bd2de3648f55" containerName="istio-proxy" Nov 03 21:19:17.007902 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.007909 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a14fb4-c56f-4704-91c9-bd2de3648f55" containerName="istio-proxy" Nov 03 21:19:17.008865 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.007920 5615 cpu_manager.go:414] "containerMap: removing container" podUID="e3271312-1e59-4a84-8ac1-d8e57f69f85b" containerName="istio-proxy" Nov 03 21:19:17.008865 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.007927 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3271312-1e59-4a84-8ac1-d8e57f69f85b" containerName="istio-proxy" Nov 03 21:19:17.008865 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.008061 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="c39084a3-cdf2-47b8-8aa1-b09415e6a90e" containerName="istio-proxy" Nov 03 21:19:17.008865 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.008097 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebbedd3d-03f0-4518-b13b-fd88e4c633d5" containerName="istio-proxy" Nov 03 21:19:17.008865 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.008113 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="11b3bf8b-24eb-4b93-80a1-fabdf14cff21" containerName="istio-proxy" Nov 03 21:19:17.008865 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.008123 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c156401-fe0c-4439-b1a8-7c9bdd23a624" containerName="maas-api" Nov 03 21:19:17.008865 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.008132 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a14fb4-c56f-4704-91c9-bd2de3648f55" containerName="istio-proxy" Nov 03 21:19:17.008865 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.008141 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3271312-1e59-4a84-8ac1-d8e57f69f85b" containerName="istio-proxy" Nov 03 21:19:17.008865 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.008160 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d82588a-f1d2-424e-b718-081993dc868c" containerName="payload-pre-processing" Nov 03 21:19:17.008865 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.008172 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="81109a63-4170-413c-bad7-4b49eaf27f0e" containerName="payload-processing" Nov 03 21:19:17.012032 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.012003 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:17.126101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.111818 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f285754f-0207-4721-a4f6-62ebfe9a2223-istiod-ca-cert\") pod \"e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:17.318425 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.318307 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f285754f-0207-4721-a4f6-62ebfe9a2223-istio-token\") pod \"e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:17.318615 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.318438 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-istio-data\") pod \"e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:17.318675 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.318610 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-istio-envoy\") pod \"e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:17.318756 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.318719 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-workload-certs\") pod \"e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:17.318823 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.318778 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-workload-socket\") pod \"e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:17.318880 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.318838 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f285754f-0207-4721-a4f6-62ebfe9a2223-istio-podinfo\") pod \"e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:17.318954 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.318934 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-credential-socket\") pod \"e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:17.319013 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.318997 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd29t\" (UniqueName: \"kubernetes.io/projected/f285754f-0207-4721-a4f6-62ebfe9a2223-kube-api-access-xd29t\") pod \"e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:17.319464 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.129128 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj"] Nov 03 21:19:17.319591 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.298685 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a77060-openshift-default-dockercfg-7fvrz\"" Nov 03 21:19:17.420258 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.420217 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f285754f-0207-4721-a4f6-62ebfe9a2223-istio-token\") pod \"e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:17.420460 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.420283 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-istio-data\") pod \"e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:17.420460 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.420386 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-istio-envoy\") pod \"e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:17.420460 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.420453 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-workload-certs\") pod \"e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:17.420645 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.420486 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-workload-socket\") pod \"e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:17.420645 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.420516 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f285754f-0207-4721-a4f6-62ebfe9a2223-istio-podinfo\") pod \"e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:17.420645 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.420577 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-credential-socket\") pod \"e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:17.420645 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.420601 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd29t\" (UniqueName: \"kubernetes.io/projected/f285754f-0207-4721-a4f6-62ebfe9a2223-kube-api-access-xd29t\") pod \"e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:17.420861 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.420697 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f285754f-0207-4721-a4f6-62ebfe9a2223-istiod-ca-cert\") pod \"e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:17.420861 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.420737 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-istio-data\") pod \"e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:17.420969 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.420951 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-workload-socket\") pod \"e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:17.421337 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.421312 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-credential-socket\") pod \"e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:17.421621 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.421596 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-workload-certs\") pod \"e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:17.421984 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.421958 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f285754f-0207-4721-a4f6-62ebfe9a2223-istiod-ca-cert\") pod \"e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:17.423627 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.423598 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-istio-envoy\") pod \"e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:17.423743 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.423686 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f285754f-0207-4721-a4f6-62ebfe9a2223-istio-podinfo\") pod \"e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:17.478822 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.478784 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f285754f-0207-4721-a4f6-62ebfe9a2223-istio-token\") pod \"e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:17.478993 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.478933 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd29t\" (UniqueName: \"kubernetes.io/projected/f285754f-0207-4721-a4f6-62ebfe9a2223-kube-api-access-xd29t\") pod \"e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:17.612259 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:17.612175 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:18.073325 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:18.073094 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj"] Nov 03 21:19:18.099866 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:18.099807 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:19:18.099969 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:18.099926 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:19:18.100009 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:18.099977 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:19:18.791674 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:18.791631 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" event={"ID":"f285754f-0207-4721-a4f6-62ebfe9a2223","Type":"ContainerStarted","Data":"cb2e92e32076d88c076279d331ee8ea131a21a8f0806c0b6eafe04c27e40fdcf"} Nov 03 21:19:18.791871 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:18.791685 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" event={"ID":"f285754f-0207-4721-a4f6-62ebfe9a2223","Type":"ContainerStarted","Data":"ba6a41cf486fab1605cde6516f5fd15af75a27ce77cfffd90cedc1c508521911"} Nov 03 21:19:18.909823 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:18.909765 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" podStartSLOduration=2.90975083 podStartE2EDuration="2.90975083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:18.908784802 +0000 UTC m=+1853.376563526" watchObservedRunningTime="2026-11-03 21:19:18.90975083 +0000 UTC m=+1853.377529574" Nov 03 21:19:19.612677 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:19.612640 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:19.618483 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:19.618455 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:19.798995 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:19.798960 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:19.800126 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:19.800063 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:19:22.167295 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:22.167252 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229"] Nov 03 21:19:22.167749 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:22.167506 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" podUID="a0bf7948-0c3d-4cae-bd74-fa0f85d51573" containerName="istio-proxy" containerID="cri-o://a0e2c4272ebab31b5e80cd379d1d2a8282958174682321844137f2bb1065bd3a" gracePeriod=30 Nov 03 21:19:22.636393 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:22.636359 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498"] Nov 03 21:19:22.637088 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:22.637054 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="11b3bf8b-24eb-4b93-80a1-fabdf14cff21" containerName="istio-proxy" Nov 03 21:19:22.637221 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:22.637095 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="c39084a3-cdf2-47b8-8aa1-b09415e6a90e" containerName="istio-proxy" Nov 03 21:19:22.640691 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:22.640670 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498" Nov 03 21:19:22.692915 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:22.692879 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-812ef9-serving-cert\"" Nov 03 21:19:22.696239 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:22.696208 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498"] Nov 03 21:19:22.713397 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:22.713363 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d28992b-9e00-4787-aa1c-3af045d450ea-maas-api-tls\") pod \"maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498\" (UID: \"2d28992b-9e00-4787-aa1c-3af045d450ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498" Nov 03 21:19:22.713576 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:22.713431 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d28992b-9e00-4787-aa1c-3af045d450ea-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498\" (UID: \"2d28992b-9e00-4787-aa1c-3af045d450ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498" Nov 03 21:19:22.713636 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:22.713612 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkdwj\" (UniqueName: \"kubernetes.io/projected/2d28992b-9e00-4787-aa1c-3af045d450ea-kube-api-access-hkdwj\") pod \"maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498\" (UID: \"2d28992b-9e00-4787-aa1c-3af045d450ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498" Nov 03 21:19:22.713701 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:22.713688 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d28992b-9e00-4787-aa1c-3af045d450ea-metrics-certs\") pod \"maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498\" (UID: \"2d28992b-9e00-4787-aa1c-3af045d450ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498" Nov 03 21:19:22.814553 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:22.814516 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d28992b-9e00-4787-aa1c-3af045d450ea-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498\" (UID: \"2d28992b-9e00-4787-aa1c-3af045d450ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498" Nov 03 21:19:22.814741 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:22.814641 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkdwj\" (UniqueName: \"kubernetes.io/projected/2d28992b-9e00-4787-aa1c-3af045d450ea-kube-api-access-hkdwj\") pod \"maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498\" (UID: \"2d28992b-9e00-4787-aa1c-3af045d450ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498" Nov 03 21:19:22.814741 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:22.814699 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d28992b-9e00-4787-aa1c-3af045d450ea-metrics-certs\") pod \"maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498\" (UID: \"2d28992b-9e00-4787-aa1c-3af045d450ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498" Nov 03 21:19:22.814831 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:22.814754 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d28992b-9e00-4787-aa1c-3af045d450ea-maas-api-tls\") pod \"maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498\" (UID: \"2d28992b-9e00-4787-aa1c-3af045d450ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498" Nov 03 21:19:22.814896 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:19:22.814879 5615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-serving-cert: secret "maas-api-e2e-ipp-b-812ef9-serving-cert" not found Nov 03 21:19:22.814970 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:19:22.814959 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d28992b-9e00-4787-aa1c-3af045d450ea-maas-api-tls podName:2d28992b-9e00-4787-aa1c-3af045d450ea nodeName:}" failed. No retries permitted until 2026-11-03 21:19:23.314941542 +0000 UTC m=+1857.782720263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2d28992b-9e00-4787-aa1c-3af045d450ea-maas-api-tls") pod "maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498" (UID: "2d28992b-9e00-4787-aa1c-3af045d450ea") : secret "maas-api-e2e-ipp-b-812ef9-serving-cert" not found Nov 03 21:19:22.815396 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:22.815367 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d28992b-9e00-4787-aa1c-3af045d450ea-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498\" (UID: \"2d28992b-9e00-4787-aa1c-3af045d450ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498" Nov 03 21:19:22.820533 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:22.817337 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d28992b-9e00-4787-aa1c-3af045d450ea-metrics-certs\") pod \"maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498\" (UID: \"2d28992b-9e00-4787-aa1c-3af045d450ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498" Nov 03 21:19:22.901815 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:22.901716 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkdwj\" (UniqueName: \"kubernetes.io/projected/2d28992b-9e00-4787-aa1c-3af045d450ea-kube-api-access-hkdwj\") pod \"maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498\" (UID: \"2d28992b-9e00-4787-aa1c-3af045d450ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498" Nov 03 21:19:23.330371 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:23.330335 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d28992b-9e00-4787-aa1c-3af045d450ea-maas-api-tls\") pod \"maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498\" (UID: \"2d28992b-9e00-4787-aa1c-3af045d450ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498" Nov 03 21:19:23.332865 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:23.332840 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d28992b-9e00-4787-aa1c-3af045d450ea-maas-api-tls\") pod \"maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498\" (UID: \"2d28992b-9e00-4787-aa1c-3af045d450ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498" Nov 03 21:19:23.555490 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:23.555456 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498" Nov 03 21:19:23.928901 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:23.928872 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498"] Nov 03 21:19:23.931689 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:19:23.931651 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d28992b_9e00_4787_aa1c_3af045d450ea.slice/crio-3202a3180d312528cc2a1418524f3475f7f0664f0782f864d0510a69e9b186cb WatchSource:0}: Error finding container 3202a3180d312528cc2a1418524f3475f7f0664f0782f864d0510a69e9b186cb: Status 404 returned error can't find the container with id 3202a3180d312528cc2a1418524f3475f7f0664f0782f864d0510a69e9b186cb Nov 03 21:19:24.836765 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:24.836730 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498" event={"ID":"2d28992b-9e00-4787-aa1c-3af045d450ea","Type":"ContainerStarted","Data":"3202a3180d312528cc2a1418524f3475f7f0664f0782f864d0510a69e9b186cb"} Nov 03 21:19:25.844946 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:25.844900 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498" event={"ID":"2d28992b-9e00-4787-aa1c-3af045d450ea","Type":"ContainerStarted","Data":"933f32ef523622bc0e044c94eb8a204c72c8626a56ac1781b6f0663f9aa60eac"} Nov 03 21:19:25.845402 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:25.845032 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498" Nov 03 21:19:25.957781 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:25.957727 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498" podStartSLOduration=3.064762447 podStartE2EDuration="3.957713921s" totalImagesPullingTime="892.951474ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:25.955930024 +0000 UTC m=+1860.423708751" watchObservedRunningTime="2026-11-03 21:19:25.957713921 +0000 UTC m=+1860.425492687" Nov 03 21:19:27.443545 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.443479 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:19:27.484147 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.484057 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istio-data\") pod \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " Nov 03 21:19:27.484345 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.484180 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-kube-api-access-tn7nl\" (UniqueName: \"kubernetes.io/projected/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-kube-api-access-tn7nl\") pod \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " Nov 03 21:19:27.484345 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.484302 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-credential-socket\") pod \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " Nov 03 21:19:27.484345 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.484341 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-workload-certs\") pod \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " Nov 03 21:19:27.484536 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.484400 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istio-data" pod "a0bf7948-0c3d-4cae-bd74-fa0f85d51573" (UID: "a0bf7948-0c3d-4cae-bd74-fa0f85d51573"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:27.484536 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.484413 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istio-envoy\") pod \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " Nov 03 21:19:27.484536 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.484438 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istio-token\" (UniqueName: \"kubernetes.io/projected/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istio-token\") pod \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " Nov 03 21:19:27.484697 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.484562 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istiod-ca-cert\") pod \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " Nov 03 21:19:27.484697 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.484595 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-workload-socket\") pod \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " Nov 03 21:19:27.484697 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.484634 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istio-podinfo\") pod \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\" (UID: \"a0bf7948-0c3d-4cae-bd74-fa0f85d51573\") " Nov 03 21:19:27.484697 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.484635 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-workload-certs" pod "a0bf7948-0c3d-4cae-bd74-fa0f85d51573" (UID: "a0bf7948-0c3d-4cae-bd74-fa0f85d51573"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:27.484912 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.484844 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-workload-socket" pod "a0bf7948-0c3d-4cae-bd74-fa0f85d51573" (UID: "a0bf7948-0c3d-4cae-bd74-fa0f85d51573"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:27.485220 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.485179 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istiod-ca-cert" pod "a0bf7948-0c3d-4cae-bd74-fa0f85d51573" (UID: "a0bf7948-0c3d-4cae-bd74-fa0f85d51573"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:19:27.485334 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.485260 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:27.485334 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.485284 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:27.485334 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.485298 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:27.485741 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.485714 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-credential-socket" pod "a0bf7948-0c3d-4cae-bd74-fa0f85d51573" (UID: "a0bf7948-0c3d-4cae-bd74-fa0f85d51573"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:27.487133 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.487051 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istio-token" pod "a0bf7948-0c3d-4cae-bd74-fa0f85d51573" (UID: "a0bf7948-0c3d-4cae-bd74-fa0f85d51573"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:27.487301 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.487284 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istio-envoy" pod "a0bf7948-0c3d-4cae-bd74-fa0f85d51573" (UID: "a0bf7948-0c3d-4cae-bd74-fa0f85d51573"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:27.487368 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.487294 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istio-podinfo" pod "a0bf7948-0c3d-4cae-bd74-fa0f85d51573" (UID: "a0bf7948-0c3d-4cae-bd74-fa0f85d51573"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:19:27.487368 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.487323 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-kube-api-access-tn7nl" pod "a0bf7948-0c3d-4cae-bd74-fa0f85d51573" (UID: "a0bf7948-0c3d-4cae-bd74-fa0f85d51573"). InnerVolumeSpecName "kube-api-access-tn7nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:27.586281 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.586241 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:27.586281 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.586278 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:27.586528 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.586293 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:27.586528 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.586307 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:27.586528 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.586321 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn7nl\" (UniqueName: \"kubernetes.io/projected/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-kube-api-access-tn7nl\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:27.586528 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.586336 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0bf7948-0c3d-4cae-bd74-fa0f85d51573-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:27.859061 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.859026 5615 generic.go:356] "Generic (PLEG): container finished" podID="a0bf7948-0c3d-4cae-bd74-fa0f85d51573" containerID="a0e2c4272ebab31b5e80cd379d1d2a8282958174682321844137f2bb1065bd3a" exitCode=0 Nov 03 21:19:27.859236 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.859099 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" Nov 03 21:19:27.859236 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.859113 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" event={"ID":"a0bf7948-0c3d-4cae-bd74-fa0f85d51573","Type":"ContainerDied","Data":"a0e2c4272ebab31b5e80cd379d1d2a8282958174682321844137f2bb1065bd3a"} Nov 03 21:19:27.859236 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.859171 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229" event={"ID":"a0bf7948-0c3d-4cae-bd74-fa0f85d51573","Type":"ContainerDied","Data":"73fc7079035888199088ac03f20e0661cc4f278a9678d05392ebc915aaed4c0c"} Nov 03 21:19:27.859236 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.859193 5615 scope.go:122] "RemoveContainer" containerID="a0e2c4272ebab31b5e80cd379d1d2a8282958174682321844137f2bb1065bd3a" Nov 03 21:19:27.868945 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.868926 5615 scope.go:122] "RemoveContainer" containerID="a0e2c4272ebab31b5e80cd379d1d2a8282958174682321844137f2bb1065bd3a" Nov 03 21:19:27.869278 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:19:27.869257 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0e2c4272ebab31b5e80cd379d1d2a8282958174682321844137f2bb1065bd3a\": container with ID starting with a0e2c4272ebab31b5e80cd379d1d2a8282958174682321844137f2bb1065bd3a not found: ID does not exist" containerID="a0e2c4272ebab31b5e80cd379d1d2a8282958174682321844137f2bb1065bd3a" Nov 03 21:19:27.869336 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:27.869289 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0e2c4272ebab31b5e80cd379d1d2a8282958174682321844137f2bb1065bd3a"} err="failed to get container status \"a0e2c4272ebab31b5e80cd379d1d2a8282958174682321844137f2bb1065bd3a\": rpc error: code = NotFound desc = could not find container \"a0e2c4272ebab31b5e80cd379d1d2a8282958174682321844137f2bb1065bd3a\": container with ID starting with a0e2c4272ebab31b5e80cd379d1d2a8282958174682321844137f2bb1065bd3a not found: ID does not exist" Nov 03 21:19:28.028495 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.028456 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229"] Nov 03 21:19:28.030984 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.030954 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-7p229"] Nov 03 21:19:28.067150 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.067113 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk"] Nov 03 21:19:28.067743 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.067729 5615 cpu_manager.go:414] "containerMap: removing container" podUID="a0bf7948-0c3d-4cae-bd74-fa0f85d51573" containerName="istio-proxy" Nov 03 21:19:28.067792 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.067745 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0bf7948-0c3d-4cae-bd74-fa0f85d51573" containerName="istio-proxy" Nov 03 21:19:28.067851 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.067840 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0bf7948-0c3d-4cae-bd74-fa0f85d51573" containerName="istio-proxy" Nov 03 21:19:28.072604 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.072584 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk" Nov 03 21:19:28.122893 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.122798 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a77060\"" Nov 03 21:19:28.123060 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.122890 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a77060-dockercfg-wds4x\"" Nov 03 21:19:28.126037 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.126008 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk"] Nov 03 21:19:28.210315 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.200954 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsxcw\" (UniqueName: \"kubernetes.io/projected/bbc36145-9288-4ffe-9c5b-d836f824fd5f-kube-api-access-fsxcw\") pod \"payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk\" (UID: \"bbc36145-9288-4ffe-9c5b-d836f824fd5f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk" Nov 03 21:19:28.210315 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.201145 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc36145-9288-4ffe-9c5b-d836f824fd5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk\" (UID: \"bbc36145-9288-4ffe-9c5b-d836f824fd5f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk" Nov 03 21:19:28.255248 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.251212 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a77060-58c7974dd-mk4wt"] Nov 03 21:19:28.271929 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.270910 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-58c7974dd-mk4wt" Nov 03 21:19:28.274974 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.274750 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a77060-58c7974dd-mk4wt"] Nov 03 21:19:28.302500 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.302466 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vzxp\" (UniqueName: \"kubernetes.io/projected/26954941-5e89-4477-b16b-f59abeaf006a-kube-api-access-5vzxp\") pod \"payload-processing-e2e-ipp-b-a77060-58c7974dd-mk4wt\" (UID: \"26954941-5e89-4477-b16b-f59abeaf006a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-58c7974dd-mk4wt" Nov 03 21:19:28.302701 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.302643 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc36145-9288-4ffe-9c5b-d836f824fd5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk\" (UID: \"bbc36145-9288-4ffe-9c5b-d836f824fd5f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk" Nov 03 21:19:28.302797 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.302780 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26954941-5e89-4477-b16b-f59abeaf006a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a77060-58c7974dd-mk4wt\" (UID: \"26954941-5e89-4477-b16b-f59abeaf006a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-58c7974dd-mk4wt" Nov 03 21:19:28.302904 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.302885 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsxcw\" (UniqueName: \"kubernetes.io/projected/bbc36145-9288-4ffe-9c5b-d836f824fd5f-kube-api-access-fsxcw\") pod \"payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk\" (UID: \"bbc36145-9288-4ffe-9c5b-d836f824fd5f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk" Nov 03 21:19:28.303369 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.303347 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc36145-9288-4ffe-9c5b-d836f824fd5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk\" (UID: \"bbc36145-9288-4ffe-9c5b-d836f824fd5f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk" Nov 03 21:19:28.361523 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.361484 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsxcw\" (UniqueName: \"kubernetes.io/projected/bbc36145-9288-4ffe-9c5b-d836f824fd5f-kube-api-access-fsxcw\") pod \"payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk\" (UID: \"bbc36145-9288-4ffe-9c5b-d836f824fd5f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk" Nov 03 21:19:28.388024 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.387938 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk" Nov 03 21:19:28.403724 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.403689 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vzxp\" (UniqueName: \"kubernetes.io/projected/26954941-5e89-4477-b16b-f59abeaf006a-kube-api-access-5vzxp\") pod \"payload-processing-e2e-ipp-b-a77060-58c7974dd-mk4wt\" (UID: \"26954941-5e89-4477-b16b-f59abeaf006a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-58c7974dd-mk4wt" Nov 03 21:19:28.403854 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.403839 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26954941-5e89-4477-b16b-f59abeaf006a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a77060-58c7974dd-mk4wt\" (UID: \"26954941-5e89-4477-b16b-f59abeaf006a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-58c7974dd-mk4wt" Nov 03 21:19:28.404665 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.404640 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26954941-5e89-4477-b16b-f59abeaf006a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a77060-58c7974dd-mk4wt\" (UID: \"26954941-5e89-4477-b16b-f59abeaf006a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-58c7974dd-mk4wt" Nov 03 21:19:28.462931 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.462899 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vzxp\" (UniqueName: \"kubernetes.io/projected/26954941-5e89-4477-b16b-f59abeaf006a-kube-api-access-5vzxp\") pod \"payload-processing-e2e-ipp-b-a77060-58c7974dd-mk4wt\" (UID: \"26954941-5e89-4477-b16b-f59abeaf006a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-58c7974dd-mk4wt" Nov 03 21:19:28.594187 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.594145 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-58c7974dd-mk4wt" Nov 03 21:19:28.756505 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.756477 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk"] Nov 03 21:19:28.759275 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:19:28.758652 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc36145_9288_4ffe_9c5b_d836f824fd5f.slice/crio-558019d27bcd3f7a103065c223bb7bc631aadd1bf55024b0f753226f1176456a WatchSource:0}: Error finding container 558019d27bcd3f7a103065c223bb7bc631aadd1bf55024b0f753226f1176456a: Status 404 returned error can't find the container with id 558019d27bcd3f7a103065c223bb7bc631aadd1bf55024b0f753226f1176456a Nov 03 21:19:28.875434 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.875366 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk" event={"ID":"bbc36145-9288-4ffe-9c5b-d836f824fd5f","Type":"ContainerStarted","Data":"e968b139d78bcd38e4f51051e3cc5bd43961d678d53e2f86f86e9441ad12b9d9"} Nov 03 21:19:28.875600 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.875500 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk" Nov 03 21:19:28.875600 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.875525 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk" event={"ID":"bbc36145-9288-4ffe-9c5b-d836f824fd5f","Type":"ContainerStarted","Data":"558019d27bcd3f7a103065c223bb7bc631aadd1bf55024b0f753226f1176456a"} Nov 03 21:19:28.966792 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.966758 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a77060-58c7974dd-mk4wt"] Nov 03 21:19:28.968662 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:19:28.968627 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26954941_5e89_4477_b16b_f59abeaf006a.slice/crio-d3dd845b446b636e22a59451b8d0dd0576df228881effe2c98c8342a33d0bdad WatchSource:0}: Error finding container d3dd845b446b636e22a59451b8d0dd0576df228881effe2c98c8342a33d0bdad: Status 404 returned error can't find the container with id d3dd845b446b636e22a59451b8d0dd0576df228881effe2c98c8342a33d0bdad Nov 03 21:19:28.987802 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:28.987754 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk" podStartSLOduration=0.987738712 podStartE2EDuration="987.738712ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:28.98640954 +0000 UTC m=+1863.454188262" watchObservedRunningTime="2026-11-03 21:19:28.987738712 +0000 UTC m=+1863.455517435" Nov 03 21:19:29.653462 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:29.653427 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0bf7948-0c3d-4cae-bd74-fa0f85d51573" path="/var/lib/kubelet/pods/a0bf7948-0c3d-4cae-bd74-fa0f85d51573/volumes" Nov 03 21:19:29.886390 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:29.886346 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-58c7974dd-mk4wt" event={"ID":"26954941-5e89-4477-b16b-f59abeaf006a","Type":"ContainerStarted","Data":"5c1bb921cbb523c4b51511c287b24860f097c8b2554e77d0bdf823866d8468a1"} Nov 03 21:19:29.886555 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:29.886415 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-58c7974dd-mk4wt" event={"ID":"26954941-5e89-4477-b16b-f59abeaf006a","Type":"ContainerStarted","Data":"d3dd845b446b636e22a59451b8d0dd0576df228881effe2c98c8342a33d0bdad"} Nov 03 21:19:29.886555 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:29.886495 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-58c7974dd-mk4wt" Nov 03 21:19:29.998096 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:29.998014 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-58c7974dd-mk4wt" podStartSLOduration=1.997996037 podStartE2EDuration="1.997996037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:29.996813289 +0000 UTC m=+1864.464592013" watchObservedRunningTime="2026-11-03 21:19:29.997996037 +0000 UTC m=+1864.465774760" Nov 03 21:19:31.857503 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:31.857471 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498" Nov 03 21:19:31.878593 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:31.878546 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7"] Nov 03 21:19:31.878812 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:31.878789 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7" podUID="c1d863b8-3685-4948-b14b-9da2377616ff" containerName="maas-api" containerID="cri-o://92f01a5dfb060587f3908c2e04655cbfba637cd15cd8de2fc36ac1a1f4f7f6d9" gracePeriod=30 Nov 03 21:19:32.229662 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.229635 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7" Nov 03 21:19:32.268279 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.268248 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1d863b8-3685-4948-b14b-9da2377616ff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1d863b8-3685-4948-b14b-9da2377616ff-maas-api-tls\") pod \"c1d863b8-3685-4948-b14b-9da2377616ff\" (UID: \"c1d863b8-3685-4948-b14b-9da2377616ff\") " Nov 03 21:19:32.268459 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.268299 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d863b8-3685-4948-b14b-9da2377616ff-kube-api-access-nclpv\" (UniqueName: \"kubernetes.io/projected/c1d863b8-3685-4948-b14b-9da2377616ff-kube-api-access-nclpv\") pod \"c1d863b8-3685-4948-b14b-9da2377616ff\" (UID: \"c1d863b8-3685-4948-b14b-9da2377616ff\") " Nov 03 21:19:32.268459 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.268396 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1d863b8-3685-4948-b14b-9da2377616ff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1d863b8-3685-4948-b14b-9da2377616ff-metrics-certs\") pod \"c1d863b8-3685-4948-b14b-9da2377616ff\" (UID: \"c1d863b8-3685-4948-b14b-9da2377616ff\") " Nov 03 21:19:32.268587 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.268500 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1d863b8-3685-4948-b14b-9da2377616ff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1d863b8-3685-4948-b14b-9da2377616ff-openshift-service-ca\") pod \"c1d863b8-3685-4948-b14b-9da2377616ff\" (UID: \"c1d863b8-3685-4948-b14b-9da2377616ff\") " Nov 03 21:19:32.268895 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.268864 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d863b8-3685-4948-b14b-9da2377616ff-openshift-service-ca" pod "c1d863b8-3685-4948-b14b-9da2377616ff" (UID: "c1d863b8-3685-4948-b14b-9da2377616ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:19:32.269097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.269055 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1d863b8-3685-4948-b14b-9da2377616ff-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:32.270587 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.270563 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1d863b8-3685-4948-b14b-9da2377616ff-metrics-certs" pod "c1d863b8-3685-4948-b14b-9da2377616ff" (UID: "c1d863b8-3685-4948-b14b-9da2377616ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:19:32.270682 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.270594 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1d863b8-3685-4948-b14b-9da2377616ff-maas-api-tls" pod "c1d863b8-3685-4948-b14b-9da2377616ff" (UID: "c1d863b8-3685-4948-b14b-9da2377616ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:19:32.270720 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.270683 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d863b8-3685-4948-b14b-9da2377616ff-kube-api-access-nclpv" pod "c1d863b8-3685-4948-b14b-9da2377616ff" (UID: "c1d863b8-3685-4948-b14b-9da2377616ff"). InnerVolumeSpecName "kube-api-access-nclpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:32.301664 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.301631 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-812243-67b64449fd-ddqlz"] Nov 03 21:19:32.301877 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.301856 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-67b64449fd-ddqlz" podUID="8ceab91d-c4f1-4eb0-baf0-753f0ac73ede" containerName="payload-processing" containerID="cri-o://14cfa4c7f8f585aac3760d9bb6dde50836839e0904b6a77eb7d05dd70d81da08" gracePeriod=30 Nov 03 21:19:32.356903 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.356867 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-75b98768b4-v4jxs"] Nov 03 21:19:32.357112 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.357088 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-75b98768b4-v4jxs" podUID="e52dbc1f-a800-4c31-b7ec-01210231d982" containerName="payload-pre-processing" containerID="cri-o://0be767a38848a16c208ae24042ccbb637c1ed346002cc28904c8659df16c0ad5" gracePeriod=30 Nov 03 21:19:32.370404 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.370375 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1d863b8-3685-4948-b14b-9da2377616ff-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:32.370519 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.370407 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nclpv\" (UniqueName: \"kubernetes.io/projected/c1d863b8-3685-4948-b14b-9da2377616ff-kube-api-access-nclpv\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:32.370519 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.370420 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1d863b8-3685-4948-b14b-9da2377616ff-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:32.623331 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.623302 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-67b64449fd-ddqlz" Nov 03 21:19:32.639180 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.639151 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-75b98768b4-v4jxs" Nov 03 21:19:32.674392 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.674350 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ceab91d-c4f1-4eb0-baf0-753f0ac73ede-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ceab91d-c4f1-4eb0-baf0-753f0ac73ede-plugins-config-volume\") pod \"8ceab91d-c4f1-4eb0-baf0-753f0ac73ede\" (UID: \"8ceab91d-c4f1-4eb0-baf0-753f0ac73ede\") " Nov 03 21:19:32.674589 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.674533 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ceab91d-c4f1-4eb0-baf0-753f0ac73ede-kube-api-access-d596g\" (UniqueName: \"kubernetes.io/projected/8ceab91d-c4f1-4eb0-baf0-753f0ac73ede-kube-api-access-d596g\") pod \"8ceab91d-c4f1-4eb0-baf0-753f0ac73ede\" (UID: \"8ceab91d-c4f1-4eb0-baf0-753f0ac73ede\") " Nov 03 21:19:32.674654 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.674594 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e52dbc1f-a800-4c31-b7ec-01210231d982-kube-api-access-s8t8r\" (UniqueName: \"kubernetes.io/projected/e52dbc1f-a800-4c31-b7ec-01210231d982-kube-api-access-s8t8r\") pod \"e52dbc1f-a800-4c31-b7ec-01210231d982\" (UID: \"e52dbc1f-a800-4c31-b7ec-01210231d982\") " Nov 03 21:19:32.674719 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.674653 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e52dbc1f-a800-4c31-b7ec-01210231d982-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52dbc1f-a800-4c31-b7ec-01210231d982-plugins-config-volume\") pod \"e52dbc1f-a800-4c31-b7ec-01210231d982\" (UID: \"e52dbc1f-a800-4c31-b7ec-01210231d982\") " Nov 03 21:19:32.674816 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.674780 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ceab91d-c4f1-4eb0-baf0-753f0ac73ede-plugins-config-volume" pod "8ceab91d-c4f1-4eb0-baf0-753f0ac73ede" (UID: "8ceab91d-c4f1-4eb0-baf0-753f0ac73ede"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:19:32.675197 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.675170 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e52dbc1f-a800-4c31-b7ec-01210231d982-plugins-config-volume" pod "e52dbc1f-a800-4c31-b7ec-01210231d982" (UID: "e52dbc1f-a800-4c31-b7ec-01210231d982"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:19:32.675313 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.675293 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52dbc1f-a800-4c31-b7ec-01210231d982-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:32.675369 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.675315 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ceab91d-c4f1-4eb0-baf0-753f0ac73ede-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:32.676709 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.676689 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ceab91d-c4f1-4eb0-baf0-753f0ac73ede-kube-api-access-d596g" pod "8ceab91d-c4f1-4eb0-baf0-753f0ac73ede" (UID: "8ceab91d-c4f1-4eb0-baf0-753f0ac73ede"). InnerVolumeSpecName "kube-api-access-d596g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:32.676817 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.676797 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e52dbc1f-a800-4c31-b7ec-01210231d982-kube-api-access-s8t8r" pod "e52dbc1f-a800-4c31-b7ec-01210231d982" (UID: "e52dbc1f-a800-4c31-b7ec-01210231d982"). InnerVolumeSpecName "kube-api-access-s8t8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:32.777475 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.777421 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d596g\" (UniqueName: \"kubernetes.io/projected/8ceab91d-c4f1-4eb0-baf0-753f0ac73ede-kube-api-access-d596g\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:32.777475 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.777466 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8t8r\" (UniqueName: \"kubernetes.io/projected/e52dbc1f-a800-4c31-b7ec-01210231d982-kube-api-access-s8t8r\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:32.948286 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.948250 5615 generic.go:356] "Generic (PLEG): container finished" podID="8ceab91d-c4f1-4eb0-baf0-753f0ac73ede" containerID="14cfa4c7f8f585aac3760d9bb6dde50836839e0904b6a77eb7d05dd70d81da08" exitCode=0 Nov 03 21:19:32.948674 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.948357 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-67b64449fd-ddqlz" event={"ID":"8ceab91d-c4f1-4eb0-baf0-753f0ac73ede","Type":"ContainerDied","Data":"14cfa4c7f8f585aac3760d9bb6dde50836839e0904b6a77eb7d05dd70d81da08"} Nov 03 21:19:32.948674 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.948470 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-67b64449fd-ddqlz" Nov 03 21:19:32.948674 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.948596 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-812243-67b64449fd-ddqlz" event={"ID":"8ceab91d-c4f1-4eb0-baf0-753f0ac73ede","Type":"ContainerDied","Data":"2aa5a93c0d251b934b3f9d310b6e58f4b765d2922fc8e613505ec126010284fa"} Nov 03 21:19:32.948674 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.948638 5615 scope.go:122] "RemoveContainer" containerID="14cfa4c7f8f585aac3760d9bb6dde50836839e0904b6a77eb7d05dd70d81da08" Nov 03 21:19:32.952266 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.952239 5615 generic.go:356] "Generic (PLEG): container finished" podID="c1d863b8-3685-4948-b14b-9da2377616ff" containerID="92f01a5dfb060587f3908c2e04655cbfba637cd15cd8de2fc36ac1a1f4f7f6d9" exitCode=0 Nov 03 21:19:32.952347 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.952324 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7" event={"ID":"c1d863b8-3685-4948-b14b-9da2377616ff","Type":"ContainerDied","Data":"92f01a5dfb060587f3908c2e04655cbfba637cd15cd8de2fc36ac1a1f4f7f6d9"} Nov 03 21:19:32.952383 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.952365 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7" event={"ID":"c1d863b8-3685-4948-b14b-9da2377616ff","Type":"ContainerDied","Data":"a95adddf1982cc7ea521107bd336d7495ab7c3bfbd4fe7bf74f421ed3edba25d"} Nov 03 21:19:32.952471 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.952458 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7" Nov 03 21:19:32.954823 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.954797 5615 generic.go:356] "Generic (PLEG): container finished" podID="e52dbc1f-a800-4c31-b7ec-01210231d982" containerID="0be767a38848a16c208ae24042ccbb637c1ed346002cc28904c8659df16c0ad5" exitCode=0 Nov 03 21:19:32.954922 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.954849 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-75b98768b4-v4jxs" Nov 03 21:19:32.954922 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.954905 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-75b98768b4-v4jxs" event={"ID":"e52dbc1f-a800-4c31-b7ec-01210231d982","Type":"ContainerDied","Data":"0be767a38848a16c208ae24042ccbb637c1ed346002cc28904c8659df16c0ad5"} Nov 03 21:19:32.955013 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.954960 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-75b98768b4-v4jxs" event={"ID":"e52dbc1f-a800-4c31-b7ec-01210231d982","Type":"ContainerDied","Data":"5af7e75aaa47ee851e2546627622c82ade59a64e5b482bb374ef0de1ed50f8e9"} Nov 03 21:19:32.962343 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.962319 5615 scope.go:122] "RemoveContainer" containerID="14cfa4c7f8f585aac3760d9bb6dde50836839e0904b6a77eb7d05dd70d81da08" Nov 03 21:19:32.962655 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:19:32.962634 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14cfa4c7f8f585aac3760d9bb6dde50836839e0904b6a77eb7d05dd70d81da08\": container with ID starting with 14cfa4c7f8f585aac3760d9bb6dde50836839e0904b6a77eb7d05dd70d81da08 not found: ID does not exist" containerID="14cfa4c7f8f585aac3760d9bb6dde50836839e0904b6a77eb7d05dd70d81da08" Nov 03 21:19:32.962746 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.962668 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14cfa4c7f8f585aac3760d9bb6dde50836839e0904b6a77eb7d05dd70d81da08"} err="failed to get container status \"14cfa4c7f8f585aac3760d9bb6dde50836839e0904b6a77eb7d05dd70d81da08\": rpc error: code = NotFound desc = could not find container \"14cfa4c7f8f585aac3760d9bb6dde50836839e0904b6a77eb7d05dd70d81da08\": container with ID starting with 14cfa4c7f8f585aac3760d9bb6dde50836839e0904b6a77eb7d05dd70d81da08 not found: ID does not exist" Nov 03 21:19:32.962746 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.962691 5615 scope.go:122] "RemoveContainer" containerID="92f01a5dfb060587f3908c2e04655cbfba637cd15cd8de2fc36ac1a1f4f7f6d9" Nov 03 21:19:32.972971 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.972946 5615 scope.go:122] "RemoveContainer" containerID="92f01a5dfb060587f3908c2e04655cbfba637cd15cd8de2fc36ac1a1f4f7f6d9" Nov 03 21:19:32.973309 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:19:32.973289 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92f01a5dfb060587f3908c2e04655cbfba637cd15cd8de2fc36ac1a1f4f7f6d9\": container with ID starting with 92f01a5dfb060587f3908c2e04655cbfba637cd15cd8de2fc36ac1a1f4f7f6d9 not found: ID does not exist" containerID="92f01a5dfb060587f3908c2e04655cbfba637cd15cd8de2fc36ac1a1f4f7f6d9" Nov 03 21:19:32.973367 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.973316 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92f01a5dfb060587f3908c2e04655cbfba637cd15cd8de2fc36ac1a1f4f7f6d9"} err="failed to get container status \"92f01a5dfb060587f3908c2e04655cbfba637cd15cd8de2fc36ac1a1f4f7f6d9\": rpc error: code = NotFound desc = could not find container \"92f01a5dfb060587f3908c2e04655cbfba637cd15cd8de2fc36ac1a1f4f7f6d9\": container with ID starting with 92f01a5dfb060587f3908c2e04655cbfba637cd15cd8de2fc36ac1a1f4f7f6d9 not found: ID does not exist" Nov 03 21:19:32.973367 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.973332 5615 scope.go:122] "RemoveContainer" containerID="0be767a38848a16c208ae24042ccbb637c1ed346002cc28904c8659df16c0ad5" Nov 03 21:19:32.983978 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.983959 5615 scope.go:122] "RemoveContainer" containerID="0be767a38848a16c208ae24042ccbb637c1ed346002cc28904c8659df16c0ad5" Nov 03 21:19:32.984308 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:19:32.984287 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0be767a38848a16c208ae24042ccbb637c1ed346002cc28904c8659df16c0ad5\": container with ID starting with 0be767a38848a16c208ae24042ccbb637c1ed346002cc28904c8659df16c0ad5 not found: ID does not exist" containerID="0be767a38848a16c208ae24042ccbb637c1ed346002cc28904c8659df16c0ad5" Nov 03 21:19:32.984390 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:32.984318 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0be767a38848a16c208ae24042ccbb637c1ed346002cc28904c8659df16c0ad5"} err="failed to get container status \"0be767a38848a16c208ae24042ccbb637c1ed346002cc28904c8659df16c0ad5\": rpc error: code = NotFound desc = could not find container \"0be767a38848a16c208ae24042ccbb637c1ed346002cc28904c8659df16c0ad5\": container with ID starting with 0be767a38848a16c208ae24042ccbb637c1ed346002cc28904c8659df16c0ad5 not found: ID does not exist" Nov 03 21:19:33.106097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:33.106006 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-75b98768b4-v4jxs"] Nov 03 21:19:33.111337 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:33.111300 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-75b98768b4-v4jxs"] Nov 03 21:19:33.213347 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:33.213310 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-812243-67b64449fd-ddqlz"] Nov 03 21:19:33.219210 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:33.219181 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-812243-67b64449fd-ddqlz"] Nov 03 21:19:33.323342 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:33.323299 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7"] Nov 03 21:19:33.327116 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:33.327070 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-812243-69c6d879bd-7mxx7"] Nov 03 21:19:33.652690 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:33.652657 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ceab91d-c4f1-4eb0-baf0-753f0ac73ede" path="/var/lib/kubelet/pods/8ceab91d-c4f1-4eb0-baf0-753f0ac73ede/volumes" Nov 03 21:19:33.653150 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:33.653131 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1d863b8-3685-4948-b14b-9da2377616ff" path="/var/lib/kubelet/pods/c1d863b8-3685-4948-b14b-9da2377616ff/volumes" Nov 03 21:19:33.653509 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:33.653497 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e52dbc1f-a800-4c31-b7ec-01210231d982" path="/var/lib/kubelet/pods/e52dbc1f-a800-4c31-b7ec-01210231d982/volumes" Nov 03 21:19:36.084016 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.083983 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg"] Nov 03 21:19:36.084758 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.084737 5615 cpu_manager.go:414] "containerMap: removing container" podUID="8ceab91d-c4f1-4eb0-baf0-753f0ac73ede" containerName="payload-processing" Nov 03 21:19:36.084823 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.084759 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ceab91d-c4f1-4eb0-baf0-753f0ac73ede" containerName="payload-processing" Nov 03 21:19:36.084823 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.084768 5615 cpu_manager.go:414] "containerMap: removing container" podUID="c1d863b8-3685-4948-b14b-9da2377616ff" containerName="maas-api" Nov 03 21:19:36.084823 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.084774 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1d863b8-3685-4948-b14b-9da2377616ff" containerName="maas-api" Nov 03 21:19:36.084823 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.084786 5615 cpu_manager.go:414] "containerMap: removing container" podUID="e52dbc1f-a800-4c31-b7ec-01210231d982" containerName="payload-pre-processing" Nov 03 21:19:36.084823 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.084793 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e52dbc1f-a800-4c31-b7ec-01210231d982" containerName="payload-pre-processing" Nov 03 21:19:36.085002 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.084939 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ceab91d-c4f1-4eb0-baf0-753f0ac73ede" containerName="payload-processing" Nov 03 21:19:36.085002 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.084949 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1d863b8-3685-4948-b14b-9da2377616ff" containerName="maas-api" Nov 03 21:19:36.085002 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.084957 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="e52dbc1f-a800-4c31-b7ec-01210231d982" containerName="payload-pre-processing" Nov 03 21:19:36.089834 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.089812 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:36.131496 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.131462 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g"] Nov 03 21:19:36.135476 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.135453 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj"] Nov 03 21:19:36.135599 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.135573 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:36.145163 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.145132 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:36.183458 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.183421 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg"] Nov 03 21:19:36.230322 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.230279 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj"] Nov 03 21:19:36.233020 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.232985 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g"] Nov 03 21:19:36.265654 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.265613 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-6fs4g\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:36.265816 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.265736 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-t7wzg\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:36.265816 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.265793 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-6fs4g\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:36.265946 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.265832 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b387372c-edf1-4d12-8460-18d61fa396ba-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-6fs4g\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:36.265946 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.265888 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-t7wzg\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:36.266014 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.265960 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-6fs4g\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:36.266014 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.265983 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-t7wzg\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:36.266014 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.266000 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9qjbj\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:36.266143 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.266025 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9qjbj\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:36.266143 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.266040 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9qjbj\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:36.266143 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.266058 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-6fs4g\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:36.266143 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.266105 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9qjbj\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:36.266260 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.266161 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dckxc\" (UniqueName: \"kubernetes.io/projected/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-kube-api-access-dckxc\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-t7wzg\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:36.266260 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.266178 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9qjbj\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:36.266260 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.266200 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9qjbj\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:36.266260 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.266230 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-t7wzg\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:36.266260 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.266250 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b387372c-edf1-4d12-8460-18d61fa396ba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-6fs4g\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:36.266412 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.266264 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b387372c-edf1-4d12-8460-18d61fa396ba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-6fs4g\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:36.266412 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.266279 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-t7wzg\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:36.266412 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.266299 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9qjbj\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:36.266412 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.266324 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-t7wzg\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:36.266412 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.266345 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9qjbj\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:36.266412 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.266369 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sswv\" (UniqueName: \"kubernetes.io/projected/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-kube-api-access-9sswv\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9qjbj\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:36.266412 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.266403 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5cf5\" (UniqueName: \"kubernetes.io/projected/b387372c-edf1-4d12-8460-18d61fa396ba-kube-api-access-d5cf5\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-6fs4g\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:36.266624 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.266424 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-t7wzg\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:36.266624 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.266440 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-6fs4g\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:36.266624 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.266476 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-t7wzg\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:36.367322 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.367234 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-6fs4g\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:36.367322 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.367317 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-t7wzg\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:36.367512 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.367439 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-6fs4g\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:36.367512 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.367495 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-t7wzg\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:36.367605 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.367517 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-6fs4g\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:36.367605 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.367552 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b387372c-edf1-4d12-8460-18d61fa396ba-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-6fs4g\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:36.367709 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.367612 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-6fs4g\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:36.367709 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.367658 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-t7wzg\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:36.367818 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.367725 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-6fs4g\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:36.367818 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.367749 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-t7wzg\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:36.367818 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.367777 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9qjbj\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:36.367979 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.367821 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-6fs4g\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:36.367979 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.367825 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9qjbj\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:36.367979 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.367876 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9qjbj\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:36.367979 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.367909 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-6fs4g\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:36.367979 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.367915 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-t7wzg\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:36.367979 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.367948 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9qjbj\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:36.368317 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.368022 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-6fs4g\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:36.368317 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.368041 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dckxc\" (UniqueName: \"kubernetes.io/projected/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-kube-api-access-dckxc\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-t7wzg\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:36.368317 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.368069 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9qjbj\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:36.368317 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.368136 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-t7wzg\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:36.368317 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.368142 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9qjbj\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:36.368317 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.368221 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-t7wzg\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:36.368317 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.368241 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-t7wzg\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:36.368317 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.368257 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b387372c-edf1-4d12-8460-18d61fa396ba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-6fs4g\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:36.368317 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.368283 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b387372c-edf1-4d12-8460-18d61fa396ba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-6fs4g\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:36.368317 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.368307 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-t7wzg\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:36.368824 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.368337 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9qjbj\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:36.368824 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.368371 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9qjbj\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:36.368824 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.368379 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-t7wzg\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:36.368824 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.368403 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9qjbj\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:36.368824 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.368427 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9qjbj\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:36.368824 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.368469 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sswv\" (UniqueName: \"kubernetes.io/projected/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-kube-api-access-9sswv\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9qjbj\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:36.368824 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.368521 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9qjbj\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:36.368824 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.368648 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-t7wzg\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:36.369336 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.368867 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9qjbj\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:36.369336 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.368526 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5cf5\" (UniqueName: \"kubernetes.io/projected/b387372c-edf1-4d12-8460-18d61fa396ba-kube-api-access-d5cf5\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-6fs4g\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:36.369336 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.368974 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-t7wzg\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:36.369336 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.369206 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-6fs4g\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:36.369933 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.369881 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-t7wzg\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:36.370654 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.370351 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-6fs4g\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:36.370654 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.370584 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9qjbj\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:36.370807 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.370714 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-t7wzg\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:36.370807 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.370724 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b387372c-edf1-4d12-8460-18d61fa396ba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-6fs4g\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:36.370807 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.370762 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9qjbj\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:36.371512 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.371484 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9qjbj\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:36.371726 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.371703 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-t7wzg\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:36.372385 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.372362 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b387372c-edf1-4d12-8460-18d61fa396ba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-6fs4g\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:36.425759 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.425722 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b387372c-edf1-4d12-8460-18d61fa396ba-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-6fs4g\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:36.426700 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.426673 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dckxc\" (UniqueName: \"kubernetes.io/projected/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-kube-api-access-dckxc\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-t7wzg\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:36.427032 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.427008 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-t7wzg\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:36.427114 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.427052 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5cf5\" (UniqueName: \"kubernetes.io/projected/b387372c-edf1-4d12-8460-18d61fa396ba-kube-api-access-d5cf5\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-6fs4g\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:36.427473 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.427453 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9qjbj\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:36.427630 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.427612 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sswv\" (UniqueName: \"kubernetes.io/projected/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-kube-api-access-9sswv\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9qjbj\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:36.456555 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.456514 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:36.464146 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.464119 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:36.703782 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.703686 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:36.885026 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.884990 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g"] Nov 03 21:19:36.889659 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.889626 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj"] Nov 03 21:19:36.889947 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.889906 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:19:36.890057 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:19:36.889977 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3a64ee6_7833_4ab0_b8cb_ee371e0a972e.slice/crio-a73ae5943c14d19d46185e3f90c98829a3b60f71534f8070b28fcaa14cfde2e8 WatchSource:0}: Error finding container a73ae5943c14d19d46185e3f90c98829a3b60f71534f8070b28fcaa14cfde2e8: Status 404 returned error can't find the container with id a73ae5943c14d19d46185e3f90c98829a3b60f71534f8070b28fcaa14cfde2e8 Nov 03 21:19:36.890057 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.889985 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:19:36.890057 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.890033 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:19:36.892451 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.892424 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:19:36.892540 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.892479 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:19:36.892540 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.892507 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:19:36.992960 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.992692 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" event={"ID":"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e","Type":"ContainerStarted","Data":"58d6569a234383c78b2774939e2adeffa9788e09c8275ae7d0eb23fee7ed668b"} Nov 03 21:19:36.992960 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.992744 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" event={"ID":"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e","Type":"ContainerStarted","Data":"a73ae5943c14d19d46185e3f90c98829a3b60f71534f8070b28fcaa14cfde2e8"} Nov 03 21:19:36.997188 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.997136 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" event={"ID":"b387372c-edf1-4d12-8460-18d61fa396ba","Type":"ContainerStarted","Data":"e904b40f35defab7aff6a9eb816c32ab891b055d1bd217ada58c5b15b179ba42"} Nov 03 21:19:36.997362 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:36.997251 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" event={"ID":"b387372c-edf1-4d12-8460-18d61fa396ba","Type":"ContainerStarted","Data":"63ff54653131f55a3e9f7613255af7c83315233e99a4ac21cf72fb941666e057"} Nov 03 21:19:37.078702 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:37.078667 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg"] Nov 03 21:19:37.080945 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:19:37.080909 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod124fd847_9e0a_45d1_ae6f_d3fdc6f62433.slice/crio-3382e6815bdcb5c2a4fc68b97a0810af6cb4282062e21a2d51101940934d7785 WatchSource:0}: Error finding container 3382e6815bdcb5c2a4fc68b97a0810af6cb4282062e21a2d51101940934d7785: Status 404 returned error can't find the container with id 3382e6815bdcb5c2a4fc68b97a0810af6cb4282062e21a2d51101940934d7785 Nov 03 21:19:37.083681 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:37.083646 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:19:37.083773 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:37.083720 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:19:37.083773 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:37.083750 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:19:37.114385 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:37.114337 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" podStartSLOduration=1.114322087 podStartE2EDuration="1.114322087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:37.111343157 +0000 UTC m=+1871.579121872" watchObservedRunningTime="2026-11-03 21:19:37.114322087 +0000 UTC m=+1871.582100807" Nov 03 21:19:37.230878 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:37.230487 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" podStartSLOduration=1.230466886 podStartE2EDuration="1.230466886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:37.226375523 +0000 UTC m=+1871.694154246" watchObservedRunningTime="2026-11-03 21:19:37.230466886 +0000 UTC m=+1871.698245609" Nov 03 21:19:37.457113 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:37.457047 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:37.458565 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:37.458537 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 21:19:37.458657 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:37.458590 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" podUID="b387372c-edf1-4d12-8460-18d61fa396ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 21:19:37.464766 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:37.464738 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:37.466067 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:37.466039 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:19:37.466195 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:37.466101 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" podUID="b3a64ee6-7833-4ab0-b8cb-ee371e0a972e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 21:19:37.977165 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:37.976627 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n"] Nov 03 21:19:37.984396 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:37.984364 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n" Nov 03 21:19:38.074097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:38.073789 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-3e1b49-serving-cert\"" Nov 03 21:19:38.098098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:38.093123 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n"] Nov 03 21:19:38.116162 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:38.110998 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d22b9636-a144-4ba6-b538-d22b253c8908-metrics-certs\") pod \"maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n\" (UID: \"d22b9636-a144-4ba6-b538-d22b253c8908\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n" Nov 03 21:19:38.116618 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:38.116572 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d22b9636-a144-4ba6-b538-d22b253c8908-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n\" (UID: \"d22b9636-a144-4ba6-b538-d22b253c8908\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n" Nov 03 21:19:38.137174 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:38.116737 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62q2t\" (UniqueName: \"kubernetes.io/projected/d22b9636-a144-4ba6-b538-d22b253c8908-kube-api-access-62q2t\") pod \"maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n\" (UID: \"d22b9636-a144-4ba6-b538-d22b253c8908\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n" Nov 03 21:19:38.137174 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:38.116863 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d22b9636-a144-4ba6-b538-d22b253c8908-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n\" (UID: \"d22b9636-a144-4ba6-b538-d22b253c8908\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n" Nov 03 21:19:38.355106 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:38.341534 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" event={"ID":"124fd847-9e0a-45d1-ae6f-d3fdc6f62433","Type":"ContainerStarted","Data":"2ebafe5c26ba1c80310ecd328dc0700d39f1b83795051faa69c98004e818cc3a"} Nov 03 21:19:38.456098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:38.455834 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" event={"ID":"124fd847-9e0a-45d1-ae6f-d3fdc6f62433","Type":"ContainerStarted","Data":"3382e6815bdcb5c2a4fc68b97a0810af6cb4282062e21a2d51101940934d7785"} Nov 03 21:19:38.456098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:38.427936 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d22b9636-a144-4ba6-b538-d22b253c8908-metrics-certs\") pod \"maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n\" (UID: \"d22b9636-a144-4ba6-b538-d22b253c8908\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n" Nov 03 21:19:38.456098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:38.456095 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d22b9636-a144-4ba6-b538-d22b253c8908-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n\" (UID: \"d22b9636-a144-4ba6-b538-d22b253c8908\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n" Nov 03 21:19:38.456408 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:38.456289 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62q2t\" (UniqueName: \"kubernetes.io/projected/d22b9636-a144-4ba6-b538-d22b253c8908-kube-api-access-62q2t\") pod \"maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n\" (UID: \"d22b9636-a144-4ba6-b538-d22b253c8908\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n" Nov 03 21:19:38.456465 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:38.456439 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d22b9636-a144-4ba6-b538-d22b253c8908-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n\" (UID: \"d22b9636-a144-4ba6-b538-d22b253c8908\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n" Nov 03 21:19:38.497107 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:38.476409 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:19:38.497107 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:38.476480 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" podUID="b3a64ee6-7833-4ab0-b8cb-ee371e0a972e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 21:19:38.497107 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:38.476563 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 21:19:38.497107 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:38.476653 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" podUID="b387372c-edf1-4d12-8460-18d61fa396ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 21:19:38.524846 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:38.524808 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d22b9636-a144-4ba6-b538-d22b253c8908-metrics-certs\") pod \"maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n\" (UID: \"d22b9636-a144-4ba6-b538-d22b253c8908\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n" Nov 03 21:19:38.530097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:38.527193 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d22b9636-a144-4ba6-b538-d22b253c8908-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n\" (UID: \"d22b9636-a144-4ba6-b538-d22b253c8908\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n" Nov 03 21:19:38.530097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:38.527921 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" podStartSLOduration=2.527896027 podStartE2EDuration="2.527896027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:38.476054384 +0000 UTC m=+1872.943833106" watchObservedRunningTime="2026-11-03 21:19:38.527896027 +0000 UTC m=+1872.995674751" Nov 03 21:19:38.571103 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:38.568874 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62q2t\" (UniqueName: \"kubernetes.io/projected/d22b9636-a144-4ba6-b538-d22b253c8908-kube-api-access-62q2t\") pod \"maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n\" (UID: \"d22b9636-a144-4ba6-b538-d22b253c8908\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n" Nov 03 21:19:38.596214 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:38.596181 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d22b9636-a144-4ba6-b538-d22b253c8908-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n\" (UID: \"d22b9636-a144-4ba6-b538-d22b253c8908\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n" Nov 03 21:19:38.619156 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:38.609334 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n" Nov 03 21:19:38.721100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:38.706837 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:38.722527 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:38.722489 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 21:19:38.722744 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:38.722715 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" podUID="124fd847-9e0a-45d1-ae6f-d3fdc6f62433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 21:19:39.464098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:39.462415 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 21:19:39.464098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:39.462484 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" podUID="b387372c-edf1-4d12-8460-18d61fa396ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 21:19:39.523099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:39.516422 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:19:39.523099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:39.516529 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" podUID="b3a64ee6-7833-4ab0-b8cb-ee371e0a972e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 21:19:39.579764 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:39.579721 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n"] Nov 03 21:19:39.714215 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:39.714111 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 21:19:39.714549 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:39.714494 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" podUID="124fd847-9e0a-45d1-ae6f-d3fdc6f62433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 21:19:39.920111 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:39.919323 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk" Nov 03 21:19:40.479098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:40.478228 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:19:40.479098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:40.478298 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" podUID="b3a64ee6-7833-4ab0-b8cb-ee371e0a972e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 21:19:40.489139 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:40.480327 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 21:19:40.489139 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:40.480377 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" podUID="b387372c-edf1-4d12-8460-18d61fa396ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 21:19:40.585734 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:40.585681 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n" event={"ID":"d22b9636-a144-4ba6-b538-d22b253c8908","Type":"ContainerStarted","Data":"e6a187097e5c39a3336eeb9d67ed045589c8c9505f1abb47d0f8d3157a9b9813"} Nov 03 21:19:40.724103 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:40.723224 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 21:19:40.724103 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:40.723287 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" podUID="124fd847-9e0a-45d1-ae6f-d3fdc6f62433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 21:19:40.929375 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:40.929335 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-58c7974dd-mk4wt" Nov 03 21:19:41.149157 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:41.101260 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" podUID="232c2524-2557-47c5-9d80-f6235972ca83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:19:41.422984 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:19:41.422846 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:19:41.431174 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:41.428554 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" podUID="39c75c51-eff9-4cb8-88b9-618dbb2ba188" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 21:19:41.481564 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:41.480828 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:19:41.481564 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:41.480902 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" podUID="b3a64ee6-7833-4ab0-b8cb-ee371e0a972e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 21:19:41.481564 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:41.480990 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 21:19:41.481564 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:41.481010 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" podUID="b387372c-edf1-4d12-8460-18d61fa396ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 21:19:41.707624 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:41.707585 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 21:19:41.707907 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:41.707860 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" podUID="124fd847-9e0a-45d1-ae6f-d3fdc6f62433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 21:19:42.465679 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:42.464209 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 21:19:42.465679 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:42.464276 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" podUID="b387372c-edf1-4d12-8460-18d61fa396ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 21:19:42.465679 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:42.465133 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:19:42.465679 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:42.465177 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" podUID="b3a64ee6-7833-4ab0-b8cb-ee371e0a972e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 21:19:42.720158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:42.706712 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 21:19:42.720158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:42.706780 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" podUID="124fd847-9e0a-45d1-ae6f-d3fdc6f62433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 21:19:42.864795 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:42.864733 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n" event={"ID":"d22b9636-a144-4ba6-b538-d22b253c8908","Type":"ContainerStarted","Data":"78dddc2662eec18339dc157b8b562fab9323ffb48b31f2481c54abf4499bc173"} Nov 03 21:19:42.865569 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:42.865534 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n" Nov 03 21:19:43.461100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:43.460510 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 21:19:43.461100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:43.460580 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" podUID="b387372c-edf1-4d12-8460-18d61fa396ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 21:19:43.469554 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:43.469507 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:19:43.469861 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:43.469827 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" podUID="b3a64ee6-7833-4ab0-b8cb-ee371e0a972e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 21:19:43.712159 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:43.711931 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 21:19:43.712343 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:43.712167 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" podUID="124fd847-9e0a-45d1-ae6f-d3fdc6f62433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 21:19:44.457891 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:44.457846 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 21:19:44.458393 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:44.457909 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" podUID="b387372c-edf1-4d12-8460-18d61fa396ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 21:19:44.465258 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:44.465225 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:19:44.465394 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:44.465287 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" podUID="b3a64ee6-7833-4ab0-b8cb-ee371e0a972e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 21:19:44.573958 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:44.573886 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n" podStartSLOduration=6.352559173 podStartE2EDuration="7.573865929s" totalImagesPullingTime="1.221306756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:43.050336144 +0000 UTC m=+1877.518114865" watchObservedRunningTime="2026-11-03 21:19:44.573865929 +0000 UTC m=+1879.041644652" Nov 03 21:19:44.575553 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:44.575524 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh"] Nov 03 21:19:44.575813 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:44.575788 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" podUID="26f4da92-abee-4727-90cc-e9f03dadaa12" containerName="istio-proxy" containerID="cri-o://17c87f664a9084b69da98d7f242e5e174502807edca3e0e70eb9917414735f66" gracePeriod=30 Nov 03 21:19:44.712145 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:44.710423 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 21:19:44.712145 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:44.710496 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" podUID="124fd847-9e0a-45d1-ae6f-d3fdc6f62433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 21:19:45.457805 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:45.457761 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 21:19:45.457997 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:45.457838 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" podUID="b387372c-edf1-4d12-8460-18d61fa396ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 21:19:45.465190 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:45.465157 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:19:45.465298 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:45.465222 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" podUID="b3a64ee6-7833-4ab0-b8cb-ee371e0a972e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 21:19:45.704359 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:45.704318 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 21:19:45.704545 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:45.704388 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" podUID="124fd847-9e0a-45d1-ae6f-d3fdc6f62433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 21:19:47.402851 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:47.401849 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:47.402851 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:47.402621 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:47.403888 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:47.403832 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:19:47.464917 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:47.464869 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:19:47.465156 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:47.464961 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" podUID="b3a64ee6-7833-4ab0-b8cb-ee371e0a972e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:19:47.516584 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:47.516552 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:47.517363 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:47.517333 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:47.518863 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:47.518837 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:19:47.679380 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:47.679286 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:47.679906 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:47.679884 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:47.681698 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:47.681674 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:19:48.894474 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:48.894439 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n" Nov 03 21:19:50.050376 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.050334 5615 generic.go:356] "Generic (PLEG): container finished" podID="26f4da92-abee-4727-90cc-e9f03dadaa12" containerID="17c87f664a9084b69da98d7f242e5e174502807edca3e0e70eb9917414735f66" exitCode=0 Nov 03 21:19:50.051125 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.050904 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" event={"ID":"26f4da92-abee-4727-90cc-e9f03dadaa12","Type":"ContainerDied","Data":"17c87f664a9084b69da98d7f242e5e174502807edca3e0e70eb9917414735f66"} Nov 03 21:19:50.051488 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.051327 5615 scope.go:122] "RemoveContainer" containerID="3cf67f2f13c4a378830b88a229fe9a0d80b115e78e470e22613e19874b4df0fc" Nov 03 21:19:50.176487 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.176453 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:19:50.263106 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.262777 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26f4da92-abee-4727-90cc-e9f03dadaa12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f4da92-abee-4727-90cc-e9f03dadaa12-istio-podinfo\") pod \"26f4da92-abee-4727-90cc-e9f03dadaa12\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " Nov 03 21:19:50.263106 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.262954 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26f4da92-abee-4727-90cc-e9f03dadaa12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f4da92-abee-4727-90cc-e9f03dadaa12-istiod-ca-cert\") pod \"26f4da92-abee-4727-90cc-e9f03dadaa12\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " Nov 03 21:19:50.263106 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.263013 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-workload-socket\") pod \"26f4da92-abee-4727-90cc-e9f03dadaa12\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " Nov 03 21:19:50.263106 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.263067 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f4da92-abee-4727-90cc-e9f03dadaa12-istio-token\" (UniqueName: \"kubernetes.io/projected/26f4da92-abee-4727-90cc-e9f03dadaa12-istio-token\") pod \"26f4da92-abee-4727-90cc-e9f03dadaa12\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " Nov 03 21:19:50.263487 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.263195 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-istio-data\") pod \"26f4da92-abee-4727-90cc-e9f03dadaa12\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " Nov 03 21:19:50.263487 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.263245 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-workload-certs\") pod \"26f4da92-abee-4727-90cc-e9f03dadaa12\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " Nov 03 21:19:50.263487 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.263279 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-credential-socket\") pod \"26f4da92-abee-4727-90cc-e9f03dadaa12\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " Nov 03 21:19:50.263487 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.263418 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-istio-envoy\") pod \"26f4da92-abee-4727-90cc-e9f03dadaa12\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " Nov 03 21:19:50.263487 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.263481 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f4da92-abee-4727-90cc-e9f03dadaa12-kube-api-access-dwj6h\" (UniqueName: \"kubernetes.io/projected/26f4da92-abee-4727-90cc-e9f03dadaa12-kube-api-access-dwj6h\") pod \"26f4da92-abee-4727-90cc-e9f03dadaa12\" (UID: \"26f4da92-abee-4727-90cc-e9f03dadaa12\") " Nov 03 21:19:50.268103 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.264961 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-workload-certs" pod "26f4da92-abee-4727-90cc-e9f03dadaa12" (UID: "26f4da92-abee-4727-90cc-e9f03dadaa12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:50.268103 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.265609 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-credential-socket" pod "26f4da92-abee-4727-90cc-e9f03dadaa12" (UID: "26f4da92-abee-4727-90cc-e9f03dadaa12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:50.268103 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.265954 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f4da92-abee-4727-90cc-e9f03dadaa12-istiod-ca-cert" pod "26f4da92-abee-4727-90cc-e9f03dadaa12" (UID: "26f4da92-abee-4727-90cc-e9f03dadaa12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:19:50.268103 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.266225 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-istio-data" pod "26f4da92-abee-4727-90cc-e9f03dadaa12" (UID: "26f4da92-abee-4727-90cc-e9f03dadaa12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:50.268103 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.267049 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-workload-socket" pod "26f4da92-abee-4727-90cc-e9f03dadaa12" (UID: "26f4da92-abee-4727-90cc-e9f03dadaa12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:50.274585 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.274553 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26f4da92-abee-4727-90cc-e9f03dadaa12-istio-podinfo" pod "26f4da92-abee-4727-90cc-e9f03dadaa12" (UID: "26f4da92-abee-4727-90cc-e9f03dadaa12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:19:50.274767 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.274598 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f4da92-abee-4727-90cc-e9f03dadaa12-kube-api-access-dwj6h" pod "26f4da92-abee-4727-90cc-e9f03dadaa12" (UID: "26f4da92-abee-4727-90cc-e9f03dadaa12"). InnerVolumeSpecName "kube-api-access-dwj6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:50.276241 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.276209 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-istio-envoy" pod "26f4da92-abee-4727-90cc-e9f03dadaa12" (UID: "26f4da92-abee-4727-90cc-e9f03dadaa12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:50.276851 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.276815 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f4da92-abee-4727-90cc-e9f03dadaa12-istio-token" pod "26f4da92-abee-4727-90cc-e9f03dadaa12" (UID: "26f4da92-abee-4727-90cc-e9f03dadaa12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:50.368528 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.367105 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f4da92-abee-4727-90cc-e9f03dadaa12-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:50.368528 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.367151 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:50.368528 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.367168 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f4da92-abee-4727-90cc-e9f03dadaa12-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:50.368528 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.367183 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:50.368528 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.367198 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:50.368528 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.367212 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:50.368528 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.367226 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f4da92-abee-4727-90cc-e9f03dadaa12-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:50.368528 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.367241 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwj6h\" (UniqueName: \"kubernetes.io/projected/26f4da92-abee-4727-90cc-e9f03dadaa12-kube-api-access-dwj6h\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:50.368528 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:50.367258 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f4da92-abee-4727-90cc-e9f03dadaa12-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:51.059035 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:51.058992 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" event={"ID":"26f4da92-abee-4727-90cc-e9f03dadaa12","Type":"ContainerDied","Data":"f9af99ea3c12f0e7a0b1eb1a966e92d5a167d357d463cc0a261d504164b9f0e8"} Nov 03 21:19:51.059555 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:51.059059 5615 scope.go:122] "RemoveContainer" containerID="17c87f664a9084b69da98d7f242e5e174502807edca3e0e70eb9917414735f66" Nov 03 21:19:51.059555 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:51.059017 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh" Nov 03 21:19:51.269883 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:51.269846 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh"] Nov 03 21:19:51.317785 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:51.317685 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-812243-openshift-default-56c9ddbcf5-k9ljh"] Nov 03 21:19:51.657626 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:51.657526 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26f4da92-abee-4727-90cc-e9f03dadaa12" path="/var/lib/kubelet/pods/26f4da92-abee-4727-90cc-e9f03dadaa12/volumes" Nov 03 21:19:53.274309 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:53.274270 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-659dffdc8d-xp2zm"] Nov 03 21:19:53.274902 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:53.274886 5615 cpu_manager.go:414] "containerMap: removing container" podUID="26f4da92-abee-4727-90cc-e9f03dadaa12" containerName="istio-proxy" Nov 03 21:19:53.275006 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:53.274903 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f4da92-abee-4727-90cc-e9f03dadaa12" containerName="istio-proxy" Nov 03 21:19:53.275006 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:53.274926 5615 cpu_manager.go:414] "containerMap: removing container" podUID="26f4da92-abee-4727-90cc-e9f03dadaa12" containerName="istio-proxy" Nov 03 21:19:53.275006 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:53.274931 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f4da92-abee-4727-90cc-e9f03dadaa12" containerName="istio-proxy" Nov 03 21:19:53.275121 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:53.275043 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f4da92-abee-4727-90cc-e9f03dadaa12" containerName="istio-proxy" Nov 03 21:19:53.275121 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:53.275053 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f4da92-abee-4727-90cc-e9f03dadaa12" containerName="istio-proxy" Nov 03 21:19:53.279806 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:53.279783 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-659dffdc8d-xp2zm" Nov 03 21:19:53.376520 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:53.376484 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b2ad2d-dockercfg-pzcfz\"" Nov 03 21:19:53.376520 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:53.376484 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b2ad2d\"" Nov 03 21:19:53.379245 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:53.379215 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-659dffdc8d-xp2zm"] Nov 03 21:19:53.423417 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:53.423382 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b2ad2d-659dffdc8d-xp2zm\" (UID: \"bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-659dffdc8d-xp2zm" Nov 03 21:19:53.423619 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:53.423446 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmssb\" (UniqueName: \"kubernetes.io/projected/bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec-kube-api-access-kmssb\") pod \"payload-pre-processing-e2e-ipp-b-b2ad2d-659dffdc8d-xp2zm\" (UID: \"bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-659dffdc8d-xp2zm" Nov 03 21:19:53.525215 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:53.525119 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b2ad2d-659dffdc8d-xp2zm\" (UID: \"bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-659dffdc8d-xp2zm" Nov 03 21:19:53.525215 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:53.525201 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmssb\" (UniqueName: \"kubernetes.io/projected/bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec-kube-api-access-kmssb\") pod \"payload-pre-processing-e2e-ipp-b-b2ad2d-659dffdc8d-xp2zm\" (UID: \"bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-659dffdc8d-xp2zm" Nov 03 21:19:53.525901 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:53.525872 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b2ad2d-659dffdc8d-xp2zm\" (UID: \"bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-659dffdc8d-xp2zm" Nov 03 21:19:53.584023 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:53.583986 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmssb\" (UniqueName: \"kubernetes.io/projected/bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec-kube-api-access-kmssb\") pod \"payload-pre-processing-e2e-ipp-b-b2ad2d-659dffdc8d-xp2zm\" (UID: \"bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-659dffdc8d-xp2zm" Nov 03 21:19:53.593866 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:53.593838 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-659dffdc8d-xp2zm" Nov 03 21:19:53.964205 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:53.964171 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-659dffdc8d-xp2zm"] Nov 03 21:19:53.965907 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:19:53.965877 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc4ad39_23d5_4ae5_ab5b_4f42b30de1ec.slice/crio-2a8a2c77249b838dddd009be218c97a6b9fea883b21aa03bfae8a7a0b02739df WatchSource:0}: Error finding container 2a8a2c77249b838dddd009be218c97a6b9fea883b21aa03bfae8a7a0b02739df: Status 404 returned error can't find the container with id 2a8a2c77249b838dddd009be218c97a6b9fea883b21aa03bfae8a7a0b02739df Nov 03 21:19:54.051902 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:54.051865 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-df45799fd-wjqsf"] Nov 03 21:19:54.057445 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:54.057418 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-df45799fd-wjqsf" Nov 03 21:19:54.090528 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:54.090469 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-659dffdc8d-xp2zm" event={"ID":"bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec","Type":"ContainerStarted","Data":"181c0b2b40fdfbdeca72700c3583e89c33f1f7ea252fa62bffdf92502a2b8092"} Nov 03 21:19:54.090721 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:54.090589 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-659dffdc8d-xp2zm" event={"ID":"bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec","Type":"ContainerStarted","Data":"2a8a2c77249b838dddd009be218c97a6b9fea883b21aa03bfae8a7a0b02739df"} Nov 03 21:19:54.090721 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:54.090626 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-659dffdc8d-xp2zm" Nov 03 21:19:54.109603 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:54.109570 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-df45799fd-wjqsf"] Nov 03 21:19:54.132811 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:54.132766 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09aa41ef-008c-45d2-bc4c-e12f58cc83e9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b2ad2d-df45799fd-wjqsf\" (UID: \"09aa41ef-008c-45d2-bc4c-e12f58cc83e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-df45799fd-wjqsf" Nov 03 21:19:54.133052 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:54.133021 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rv7f\" (UniqueName: \"kubernetes.io/projected/09aa41ef-008c-45d2-bc4c-e12f58cc83e9-kube-api-access-4rv7f\") pod \"payload-processing-e2e-ipp-b-b2ad2d-df45799fd-wjqsf\" (UID: \"09aa41ef-008c-45d2-bc4c-e12f58cc83e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-df45799fd-wjqsf" Nov 03 21:19:54.234311 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:54.234272 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09aa41ef-008c-45d2-bc4c-e12f58cc83e9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b2ad2d-df45799fd-wjqsf\" (UID: \"09aa41ef-008c-45d2-bc4c-e12f58cc83e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-df45799fd-wjqsf" Nov 03 21:19:54.234478 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:54.234416 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rv7f\" (UniqueName: \"kubernetes.io/projected/09aa41ef-008c-45d2-bc4c-e12f58cc83e9-kube-api-access-4rv7f\") pod \"payload-processing-e2e-ipp-b-b2ad2d-df45799fd-wjqsf\" (UID: \"09aa41ef-008c-45d2-bc4c-e12f58cc83e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-df45799fd-wjqsf" Nov 03 21:19:54.235015 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:54.234994 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09aa41ef-008c-45d2-bc4c-e12f58cc83e9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b2ad2d-df45799fd-wjqsf\" (UID: \"09aa41ef-008c-45d2-bc4c-e12f58cc83e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-df45799fd-wjqsf" Nov 03 21:19:54.282689 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:54.282636 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-659dffdc8d-xp2zm" podStartSLOduration=1.282623566 podStartE2EDuration="1.282623566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:54.280976092 +0000 UTC m=+1888.748754808" watchObservedRunningTime="2026-11-03 21:19:54.282623566 +0000 UTC m=+1888.750402288" Nov 03 21:19:54.293120 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:54.293069 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rv7f\" (UniqueName: \"kubernetes.io/projected/09aa41ef-008c-45d2-bc4c-e12f58cc83e9-kube-api-access-4rv7f\") pod \"payload-processing-e2e-ipp-b-b2ad2d-df45799fd-wjqsf\" (UID: \"09aa41ef-008c-45d2-bc4c-e12f58cc83e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-df45799fd-wjqsf" Nov 03 21:19:54.373878 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:54.373839 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-df45799fd-wjqsf" Nov 03 21:19:54.806662 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:54.806626 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-df45799fd-wjqsf"] Nov 03 21:19:54.808261 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:19:54.808217 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09aa41ef_008c_45d2_bc4c_e12f58cc83e9.slice/crio-31773de6e585d32903d0cbc100edbbdccf1d08300d80cac9432d8c2db7b87aed WatchSource:0}: Error finding container 31773de6e585d32903d0cbc100edbbdccf1d08300d80cac9432d8c2db7b87aed: Status 404 returned error can't find the container with id 31773de6e585d32903d0cbc100edbbdccf1d08300d80cac9432d8c2db7b87aed Nov 03 21:19:55.099053 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:55.098961 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-df45799fd-wjqsf" event={"ID":"09aa41ef-008c-45d2-bc4c-e12f58cc83e9","Type":"ContainerStarted","Data":"f2347132c9d50f36b86b17810a6ee2e40388d7e8814fcec260be2b9f9e255f84"} Nov 03 21:19:55.099053 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:55.099023 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-df45799fd-wjqsf" event={"ID":"09aa41ef-008c-45d2-bc4c-e12f58cc83e9","Type":"ContainerStarted","Data":"31773de6e585d32903d0cbc100edbbdccf1d08300d80cac9432d8c2db7b87aed"} Nov 03 21:19:55.210156 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:55.210097 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-df45799fd-wjqsf" podStartSLOduration=1.210068033 podStartE2EDuration="1.210068033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:55.209335636 +0000 UTC m=+1889.677114361" watchObservedRunningTime="2026-11-03 21:19:55.210068033 +0000 UTC m=+1889.677846756" Nov 03 21:19:56.106315 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:19:56.106271 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-df45799fd-wjqsf" Nov 03 21:20:05.101597 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:05.101564 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-659dffdc8d-xp2zm" Nov 03 21:20:07.114908 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:07.114874 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-df45799fd-wjqsf" Nov 03 21:20:10.305650 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:10.305608 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk"] Nov 03 21:20:10.311153 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:10.311127 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk" Nov 03 21:20:10.358206 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:10.358166 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szp72\" (UniqueName: \"kubernetes.io/projected/ab30a7f6-78c8-4365-b2d5-7040cac18697-kube-api-access-szp72\") pod \"payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk\" (UID: \"ab30a7f6-78c8-4365-b2d5-7040cac18697\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk" Nov 03 21:20:10.358376 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:10.358319 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab30a7f6-78c8-4365-b2d5-7040cac18697-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk\" (UID: \"ab30a7f6-78c8-4365-b2d5-7040cac18697\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk" Nov 03 21:20:10.361606 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:10.361577 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-eae138-dockercfg-ntg7t\"" Nov 03 21:20:10.408719 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:10.408651 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-eae138\"" Nov 03 21:20:10.411532 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:10.411506 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk"] Nov 03 21:20:10.459670 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:10.459625 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szp72\" (UniqueName: \"kubernetes.io/projected/ab30a7f6-78c8-4365-b2d5-7040cac18697-kube-api-access-szp72\") pod \"payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk\" (UID: \"ab30a7f6-78c8-4365-b2d5-7040cac18697\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk" Nov 03 21:20:10.459868 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:10.459754 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab30a7f6-78c8-4365-b2d5-7040cac18697-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk\" (UID: \"ab30a7f6-78c8-4365-b2d5-7040cac18697\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk" Nov 03 21:20:10.460493 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:10.460474 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab30a7f6-78c8-4365-b2d5-7040cac18697-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk\" (UID: \"ab30a7f6-78c8-4365-b2d5-7040cac18697\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk" Nov 03 21:20:10.518293 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:10.518255 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szp72\" (UniqueName: \"kubernetes.io/projected/ab30a7f6-78c8-4365-b2d5-7040cac18697-kube-api-access-szp72\") pod \"payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk\" (UID: \"ab30a7f6-78c8-4365-b2d5-7040cac18697\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk" Nov 03 21:20:10.619321 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:20:10.619233 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:20:10.627604 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:10.627577 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk" Nov 03 21:20:10.767597 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:10.767547 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-7d548d4ccd-vfgnw"] Nov 03 21:20:10.773870 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:10.773840 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-7d548d4ccd-vfgnw" Nov 03 21:20:10.838622 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:10.838577 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-7d548d4ccd-vfgnw"] Nov 03 21:20:10.865010 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:10.864970 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjmnq\" (UniqueName: \"kubernetes.io/projected/f7915873-20d0-4151-937d-11581c1a1104-kube-api-access-sjmnq\") pod \"payload-pre-processing-e2e-ipp-a-eae138-7d548d4ccd-vfgnw\" (UID: \"f7915873-20d0-4151-937d-11581c1a1104\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-7d548d4ccd-vfgnw" Nov 03 21:20:10.865202 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:10.865165 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7915873-20d0-4151-937d-11581c1a1104-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-eae138-7d548d4ccd-vfgnw\" (UID: \"f7915873-20d0-4151-937d-11581c1a1104\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-7d548d4ccd-vfgnw" Nov 03 21:20:10.966044 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:10.965949 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7915873-20d0-4151-937d-11581c1a1104-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-eae138-7d548d4ccd-vfgnw\" (UID: \"f7915873-20d0-4151-937d-11581c1a1104\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-7d548d4ccd-vfgnw" Nov 03 21:20:10.966251 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:10.966118 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjmnq\" (UniqueName: \"kubernetes.io/projected/f7915873-20d0-4151-937d-11581c1a1104-kube-api-access-sjmnq\") pod \"payload-pre-processing-e2e-ipp-a-eae138-7d548d4ccd-vfgnw\" (UID: \"f7915873-20d0-4151-937d-11581c1a1104\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-7d548d4ccd-vfgnw" Nov 03 21:20:10.966743 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:10.966719 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7915873-20d0-4151-937d-11581c1a1104-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-eae138-7d548d4ccd-vfgnw\" (UID: \"f7915873-20d0-4151-937d-11581c1a1104\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-7d548d4ccd-vfgnw" Nov 03 21:20:10.999966 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:10.999934 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk"] Nov 03 21:20:11.001643 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:20:11.001607 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab30a7f6_78c8_4365_b2d5_7040cac18697.slice/crio-2a82aaea051f48308fdd54212ab58c791c838995046cc1b2248119d03a9c8707 WatchSource:0}: Error finding container 2a82aaea051f48308fdd54212ab58c791c838995046cc1b2248119d03a9c8707: Status 404 returned error can't find the container with id 2a82aaea051f48308fdd54212ab58c791c838995046cc1b2248119d03a9c8707 Nov 03 21:20:11.024472 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:11.024443 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjmnq\" (UniqueName: \"kubernetes.io/projected/f7915873-20d0-4151-937d-11581c1a1104-kube-api-access-sjmnq\") pod \"payload-pre-processing-e2e-ipp-a-eae138-7d548d4ccd-vfgnw\" (UID: \"f7915873-20d0-4151-937d-11581c1a1104\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-7d548d4ccd-vfgnw" Nov 03 21:20:11.093316 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:11.093281 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-7d548d4ccd-vfgnw" Nov 03 21:20:11.211397 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:11.211358 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk" event={"ID":"ab30a7f6-78c8-4365-b2d5-7040cac18697","Type":"ContainerStarted","Data":"afa04589846cf9b4f8e7008151ee233c94e36f87f86d05d2d340bda85c761c64"} Nov 03 21:20:11.211582 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:11.211428 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk" event={"ID":"ab30a7f6-78c8-4365-b2d5-7040cac18697","Type":"ContainerStarted","Data":"2a82aaea051f48308fdd54212ab58c791c838995046cc1b2248119d03a9c8707"} Nov 03 21:20:11.211633 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:11.211572 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk" Nov 03 21:20:11.325809 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:11.325746 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk" podStartSLOduration=1.325733376 podStartE2EDuration="1.325733376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:20:11.322209228 +0000 UTC m=+1905.789987951" watchObservedRunningTime="2026-11-03 21:20:11.325733376 +0000 UTC m=+1905.793512099" Nov 03 21:20:11.483932 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:11.483880 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-7d548d4ccd-vfgnw"] Nov 03 21:20:11.498611 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:20:11.496169 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7915873_20d0_4151_937d_11581c1a1104.slice/crio-c487214a7174e6e40fbd59b964cafad5d34b70bba597e6ab52658ab259b6cacc WatchSource:0}: Error finding container c487214a7174e6e40fbd59b964cafad5d34b70bba597e6ab52658ab259b6cacc: Status 404 returned error can't find the container with id c487214a7174e6e40fbd59b964cafad5d34b70bba597e6ab52658ab259b6cacc Nov 03 21:20:12.259218 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:12.259170 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-7d548d4ccd-vfgnw" event={"ID":"f7915873-20d0-4151-937d-11581c1a1104","Type":"ContainerStarted","Data":"08073f5701f4fe9f985fa8dd85e0f93266c440ec26c5e0542caf26077ded4606"} Nov 03 21:20:12.259445 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:12.259244 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-7d548d4ccd-vfgnw" event={"ID":"f7915873-20d0-4151-937d-11581c1a1104","Type":"ContainerStarted","Data":"c487214a7174e6e40fbd59b964cafad5d34b70bba597e6ab52658ab259b6cacc"} Nov 03 21:20:12.259543 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:12.259517 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-7d548d4ccd-vfgnw" Nov 03 21:20:12.372245 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:12.372187 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-7d548d4ccd-vfgnw" podStartSLOduration=2.372172332 podStartE2EDuration="2.372172332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:20:12.370115604 +0000 UTC m=+1906.837894328" watchObservedRunningTime="2026-11-03 21:20:12.372172332 +0000 UTC m=+1906.839951101" Nov 03 21:20:20.841961 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:20.841925 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq"] Nov 03 21:20:20.846163 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:20.846142 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq" Nov 03 21:20:20.895760 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:20.895726 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a77060-serving-cert\"" Nov 03 21:20:20.908554 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:20.908513 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/922f3c9b-ce05-4037-8b8c-b229f495e393-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq\" (UID: \"922f3c9b-ce05-4037-8b8c-b229f495e393\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq" Nov 03 21:20:20.908554 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:20.908557 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6266\" (UniqueName: \"kubernetes.io/projected/922f3c9b-ce05-4037-8b8c-b229f495e393-kube-api-access-h6266\") pod \"maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq\" (UID: \"922f3c9b-ce05-4037-8b8c-b229f495e393\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq" Nov 03 21:20:20.908772 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:20.908729 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/922f3c9b-ce05-4037-8b8c-b229f495e393-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq\" (UID: \"922f3c9b-ce05-4037-8b8c-b229f495e393\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq" Nov 03 21:20:20.908814 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:20.908778 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/922f3c9b-ce05-4037-8b8c-b229f495e393-metrics-certs\") pod \"maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq\" (UID: \"922f3c9b-ce05-4037-8b8c-b229f495e393\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq" Nov 03 21:20:20.945899 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:20.945861 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq"] Nov 03 21:20:21.010025 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:21.009982 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/922f3c9b-ce05-4037-8b8c-b229f495e393-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq\" (UID: \"922f3c9b-ce05-4037-8b8c-b229f495e393\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq" Nov 03 21:20:21.010228 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:21.010140 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/922f3c9b-ce05-4037-8b8c-b229f495e393-metrics-certs\") pod \"maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq\" (UID: \"922f3c9b-ce05-4037-8b8c-b229f495e393\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq" Nov 03 21:20:21.010228 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:20:21.010195 5615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-serving-cert: secret "maas-api-e2e-ipp-b-a77060-serving-cert" not found Nov 03 21:20:21.010338 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:21.010229 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/922f3c9b-ce05-4037-8b8c-b229f495e393-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq\" (UID: \"922f3c9b-ce05-4037-8b8c-b229f495e393\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq" Nov 03 21:20:21.010338 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:21.010252 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6266\" (UniqueName: \"kubernetes.io/projected/922f3c9b-ce05-4037-8b8c-b229f495e393-kube-api-access-h6266\") pod \"maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq\" (UID: \"922f3c9b-ce05-4037-8b8c-b229f495e393\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq" Nov 03 21:20:21.010338 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:20:21.010302 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/922f3c9b-ce05-4037-8b8c-b229f495e393-maas-api-tls podName:922f3c9b-ce05-4037-8b8c-b229f495e393 nodeName:}" failed. No retries permitted until 2026-11-03 21:20:21.510277955 +0000 UTC m=+1915.978056673 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/922f3c9b-ce05-4037-8b8c-b229f495e393-maas-api-tls") pod "maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq" (UID: "922f3c9b-ce05-4037-8b8c-b229f495e393") : secret "maas-api-e2e-ipp-b-a77060-serving-cert" not found Nov 03 21:20:21.011031 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:21.011006 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/922f3c9b-ce05-4037-8b8c-b229f495e393-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq\" (UID: \"922f3c9b-ce05-4037-8b8c-b229f495e393\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq" Nov 03 21:20:21.012838 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:21.012802 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/922f3c9b-ce05-4037-8b8c-b229f495e393-metrics-certs\") pod \"maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq\" (UID: \"922f3c9b-ce05-4037-8b8c-b229f495e393\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq" Nov 03 21:20:21.069283 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:21.069234 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6266\" (UniqueName: \"kubernetes.io/projected/922f3c9b-ce05-4037-8b8c-b229f495e393-kube-api-access-h6266\") pod \"maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq\" (UID: \"922f3c9b-ce05-4037-8b8c-b229f495e393\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq" Nov 03 21:20:21.534449 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:21.534411 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/922f3c9b-ce05-4037-8b8c-b229f495e393-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq\" (UID: \"922f3c9b-ce05-4037-8b8c-b229f495e393\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq" Nov 03 21:20:21.537012 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:21.536980 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/922f3c9b-ce05-4037-8b8c-b229f495e393-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq\" (UID: \"922f3c9b-ce05-4037-8b8c-b229f495e393\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq" Nov 03 21:20:21.761256 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:21.761213 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq" Nov 03 21:20:22.134589 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:22.134548 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq"] Nov 03 21:20:22.136158 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:20:22.136117 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod922f3c9b_ce05_4037_8b8c_b229f495e393.slice/crio-11a58c57b1092942698cb7880bab674da8987a34818b81a682ce183bce50ea3d WatchSource:0}: Error finding container 11a58c57b1092942698cb7880bab674da8987a34818b81a682ce183bce50ea3d: Status 404 returned error can't find the container with id 11a58c57b1092942698cb7880bab674da8987a34818b81a682ce183bce50ea3d Nov 03 21:20:22.137647 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:22.137627 5615 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:20:22.262012 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:22.261983 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk" Nov 03 21:20:22.326036 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:22.325999 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq" event={"ID":"922f3c9b-ce05-4037-8b8c-b229f495e393","Type":"ContainerStarted","Data":"11a58c57b1092942698cb7880bab674da8987a34818b81a682ce183bce50ea3d"} Nov 03 21:20:23.269240 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:23.269207 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-7d548d4ccd-vfgnw" Nov 03 21:20:23.337900 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:23.337811 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq" event={"ID":"922f3c9b-ce05-4037-8b8c-b229f495e393","Type":"ContainerStarted","Data":"e29e311d06d97edbfabfed05be64a810db5999c8bfc4e54cb913053cc67231cc"} Nov 03 21:20:23.337900 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:23.337874 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq" Nov 03 21:20:23.491543 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:23.491489 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq" podStartSLOduration=2.605082737 podStartE2EDuration="3.491475594s" totalImagesPullingTime="886.392857ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:20:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:20:23.490869799 +0000 UTC m=+1917.958648523" watchObservedRunningTime="2026-11-03 21:20:23.491475594 +0000 UTC m=+1917.959254316" Nov 03 21:20:29.350531 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:29.350491 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq" Nov 03 21:20:37.980068 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:37.980025 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9"] Nov 03 21:20:37.980941 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:37.980906 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" podUID="13f7f450-acbe-4a27-aedc-4c3f43c942e5" containerName="istio-proxy" containerID="cri-o://fa9b3391393f8f5ba0aa10203e9226dbb3562ff89fc814bd17b402d9aa6da0ff" gracePeriod=30 Nov 03 21:20:39.421469 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:39.421432 5615 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:20:39.421939 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:39.421494 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" podUID="13f7f450-acbe-4a27-aedc-4c3f43c942e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:20:40.643302 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:20:40.643261 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:20:43.401132 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.397994 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:20:43.479483 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.479450 5615 generic.go:356] "Generic (PLEG): container finished" podID="13f7f450-acbe-4a27-aedc-4c3f43c942e5" containerID="fa9b3391393f8f5ba0aa10203e9226dbb3562ff89fc814bd17b402d9aa6da0ff" exitCode=0 Nov 03 21:20:43.479662 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.479603 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" event={"ID":"13f7f450-acbe-4a27-aedc-4c3f43c942e5","Type":"ContainerDied","Data":"fa9b3391393f8f5ba0aa10203e9226dbb3562ff89fc814bd17b402d9aa6da0ff"} Nov 03 21:20:43.479718 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.479679 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" event={"ID":"13f7f450-acbe-4a27-aedc-4c3f43c942e5","Type":"ContainerDied","Data":"20a176a57d98fe83ec15c1b47a7f1f2ad89786d6a710e0da3bb360109fe09798"} Nov 03 21:20:43.479718 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.479707 5615 scope.go:122] "RemoveContainer" containerID="fa9b3391393f8f5ba0aa10203e9226dbb3562ff89fc814bd17b402d9aa6da0ff" Nov 03 21:20:43.479866 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.479850 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9" Nov 03 21:20:43.493625 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.493586 5615 scope.go:122] "RemoveContainer" containerID="fa9b3391393f8f5ba0aa10203e9226dbb3562ff89fc814bd17b402d9aa6da0ff" Nov 03 21:20:43.493900 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.493879 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-workload-certs\") pod \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " Nov 03 21:20:43.493972 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:20:43.493932 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa9b3391393f8f5ba0aa10203e9226dbb3562ff89fc814bd17b402d9aa6da0ff\": container with ID starting with fa9b3391393f8f5ba0aa10203e9226dbb3562ff89fc814bd17b402d9aa6da0ff not found: ID does not exist" containerID="fa9b3391393f8f5ba0aa10203e9226dbb3562ff89fc814bd17b402d9aa6da0ff" Nov 03 21:20:43.494029 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.493969 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa9b3391393f8f5ba0aa10203e9226dbb3562ff89fc814bd17b402d9aa6da0ff"} err="failed to get container status \"fa9b3391393f8f5ba0aa10203e9226dbb3562ff89fc814bd17b402d9aa6da0ff\": rpc error: code = NotFound desc = could not find container \"fa9b3391393f8f5ba0aa10203e9226dbb3562ff89fc814bd17b402d9aa6da0ff\": container with ID starting with fa9b3391393f8f5ba0aa10203e9226dbb3562ff89fc814bd17b402d9aa6da0ff not found: ID does not exist" Nov 03 21:20:43.494029 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.493940 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13f7f450-acbe-4a27-aedc-4c3f43c942e5-kube-api-access-qfbrn\" (UniqueName: \"kubernetes.io/projected/13f7f450-acbe-4a27-aedc-4c3f43c942e5-kube-api-access-qfbrn\") pod \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " Nov 03 21:20:43.494184 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.494145 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istiod-ca-cert\") pod \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " Nov 03 21:20:43.494230 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.494203 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istio-podinfo\") pod \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " Nov 03 21:20:43.494279 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.494171 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-workload-certs" pod "13f7f450-acbe-4a27-aedc-4c3f43c942e5" (UID: "13f7f450-acbe-4a27-aedc-4c3f43c942e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:20:43.494334 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.494290 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istio-data\") pod \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " Nov 03 21:20:43.494334 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.494322 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-workload-socket\") pod \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " Nov 03 21:20:43.494447 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.494361 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istio-token\" (UniqueName: \"kubernetes.io/projected/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istio-token\") pod \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " Nov 03 21:20:43.494519 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.494500 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istio-envoy\") pod \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " Nov 03 21:20:43.494579 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.494449 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istiod-ca-cert" pod "13f7f450-acbe-4a27-aedc-4c3f43c942e5" (UID: "13f7f450-acbe-4a27-aedc-4c3f43c942e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:20:43.494579 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.494551 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istio-data" pod "13f7f450-acbe-4a27-aedc-4c3f43c942e5" (UID: "13f7f450-acbe-4a27-aedc-4c3f43c942e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:20:43.494579 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.494568 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-workload-socket" pod "13f7f450-acbe-4a27-aedc-4c3f43c942e5" (UID: "13f7f450-acbe-4a27-aedc-4c3f43c942e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:20:43.494739 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.494572 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-credential-socket\") pod \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\" (UID: \"13f7f450-acbe-4a27-aedc-4c3f43c942e5\") " Nov 03 21:20:43.494792 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.494756 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-credential-socket" pod "13f7f450-acbe-4a27-aedc-4c3f43c942e5" (UID: "13f7f450-acbe-4a27-aedc-4c3f43c942e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:20:43.495237 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.495212 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:43.495343 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.495243 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:43.495343 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.495260 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:43.495343 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.495274 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:43.495343 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.495289 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:43.496759 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.496734 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istio-podinfo" pod "13f7f450-acbe-4a27-aedc-4c3f43c942e5" (UID: "13f7f450-acbe-4a27-aedc-4c3f43c942e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:20:43.496852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.496774 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istio-envoy" pod "13f7f450-acbe-4a27-aedc-4c3f43c942e5" (UID: "13f7f450-acbe-4a27-aedc-4c3f43c942e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:20:43.497016 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.497001 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f7f450-acbe-4a27-aedc-4c3f43c942e5-kube-api-access-qfbrn" pod "13f7f450-acbe-4a27-aedc-4c3f43c942e5" (UID: "13f7f450-acbe-4a27-aedc-4c3f43c942e5"). InnerVolumeSpecName "kube-api-access-qfbrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:20:43.524374 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.514750 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istio-token" pod "13f7f450-acbe-4a27-aedc-4c3f43c942e5" (UID: "13f7f450-acbe-4a27-aedc-4c3f43c942e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:20:43.596548 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.596455 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:43.596548 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.596490 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfbrn\" (UniqueName: \"kubernetes.io/projected/13f7f450-acbe-4a27-aedc-4c3f43c942e5-kube-api-access-qfbrn\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:43.596548 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.596502 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:43.596548 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.596512 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13f7f450-acbe-4a27-aedc-4c3f43c942e5-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:43.890680 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.890597 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9"] Nov 03 21:20:43.896068 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:43.896038 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m6hd9"] Nov 03 21:20:45.654497 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:45.654466 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13f7f450-acbe-4a27-aedc-4c3f43c942e5" path="/var/lib/kubelet/pods/13f7f450-acbe-4a27-aedc-4c3f43c942e5/volumes" Nov 03 21:20:48.612103 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:48.608793 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-df45799fd-wjqsf"] Nov 03 21:20:48.612103 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:48.609054 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-df45799fd-wjqsf" podUID="09aa41ef-008c-45d2-bc4c-e12f58cc83e9" containerName="payload-processing" containerID="cri-o://f2347132c9d50f36b86b17810a6ee2e40388d7e8814fcec260be2b9f9e255f84" gracePeriod=30 Nov 03 21:20:48.657702 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:48.657667 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-659dffdc8d-xp2zm"] Nov 03 21:20:48.657895 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:48.657857 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-659dffdc8d-xp2zm" podUID="bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec" containerName="payload-pre-processing" containerID="cri-o://181c0b2b40fdfbdeca72700c3583e89c33f1f7ea252fa62bffdf92502a2b8092" gracePeriod=30 Nov 03 21:20:48.965320 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:48.965284 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-df45799fd-wjqsf" Nov 03 21:20:48.994923 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:48.994895 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-659dffdc8d-xp2zm" Nov 03 21:20:49.117104 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.111872 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09aa41ef-008c-45d2-bc4c-e12f58cc83e9-kube-api-access-4rv7f\" (UniqueName: \"kubernetes.io/projected/09aa41ef-008c-45d2-bc4c-e12f58cc83e9-kube-api-access-4rv7f\") pod \"09aa41ef-008c-45d2-bc4c-e12f58cc83e9\" (UID: \"09aa41ef-008c-45d2-bc4c-e12f58cc83e9\") " Nov 03 21:20:49.117104 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.111962 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec-plugins-config-volume\") pod \"bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec\" (UID: \"bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec\") " Nov 03 21:20:49.117104 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.111995 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09aa41ef-008c-45d2-bc4c-e12f58cc83e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09aa41ef-008c-45d2-bc4c-e12f58cc83e9-plugins-config-volume\") pod \"09aa41ef-008c-45d2-bc4c-e12f58cc83e9\" (UID: \"09aa41ef-008c-45d2-bc4c-e12f58cc83e9\") " Nov 03 21:20:49.117104 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.112044 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec-kube-api-access-kmssb\" (UniqueName: \"kubernetes.io/projected/bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec-kube-api-access-kmssb\") pod \"bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec\" (UID: \"bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec\") " Nov 03 21:20:49.117104 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.114041 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09aa41ef-008c-45d2-bc4c-e12f58cc83e9-plugins-config-volume" pod "09aa41ef-008c-45d2-bc4c-e12f58cc83e9" (UID: "09aa41ef-008c-45d2-bc4c-e12f58cc83e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:20:49.144163 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.135099 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec-plugins-config-volume" pod "bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec" (UID: "bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:20:49.144163 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.142417 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09aa41ef-008c-45d2-bc4c-e12f58cc83e9-kube-api-access-4rv7f" pod "09aa41ef-008c-45d2-bc4c-e12f58cc83e9" (UID: "09aa41ef-008c-45d2-bc4c-e12f58cc83e9"). InnerVolumeSpecName "kube-api-access-4rv7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:20:49.144163 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.143832 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec-kube-api-access-kmssb" pod "bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec" (UID: "bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec"). InnerVolumeSpecName "kube-api-access-kmssb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:20:49.221102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.220103 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rv7f\" (UniqueName: \"kubernetes.io/projected/09aa41ef-008c-45d2-bc4c-e12f58cc83e9-kube-api-access-4rv7f\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:49.221102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.220145 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:49.221102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.220161 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09aa41ef-008c-45d2-bc4c-e12f58cc83e9-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:49.221102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.220174 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmssb\" (UniqueName: \"kubernetes.io/projected/bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec-kube-api-access-kmssb\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:49.457336 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.457240 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5"] Nov 03 21:20:49.458807 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.457960 5615 cpu_manager.go:414] "containerMap: removing container" podUID="09aa41ef-008c-45d2-bc4c-e12f58cc83e9" containerName="payload-processing" Nov 03 21:20:49.458807 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.457999 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09aa41ef-008c-45d2-bc4c-e12f58cc83e9" containerName="payload-processing" Nov 03 21:20:49.458807 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.458043 5615 cpu_manager.go:414] "containerMap: removing container" podUID="bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec" containerName="payload-pre-processing" Nov 03 21:20:49.458807 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.458049 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec" containerName="payload-pre-processing" Nov 03 21:20:49.458807 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.458065 5615 cpu_manager.go:414] "containerMap: removing container" podUID="13f7f450-acbe-4a27-aedc-4c3f43c942e5" containerName="istio-proxy" Nov 03 21:20:49.458807 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.458085 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13f7f450-acbe-4a27-aedc-4c3f43c942e5" containerName="istio-proxy" Nov 03 21:20:49.458807 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.458180 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="09aa41ef-008c-45d2-bc4c-e12f58cc83e9" containerName="payload-processing" Nov 03 21:20:49.458807 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.458195 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec" containerName="payload-pre-processing" Nov 03 21:20:49.458807 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.458202 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="13f7f450-acbe-4a27-aedc-4c3f43c942e5" containerName="istio-proxy" Nov 03 21:20:49.461377 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.461354 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:49.524099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.523476 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-7af9dd-openshift-default-dockercfg-zgkrz\"" Nov 03 21:20:49.565218 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.558049 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5"] Nov 03 21:20:49.667377 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.666100 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a41dcb03-3288-4425-866e-4473c432b8b5-istiod-ca-cert\") pod \"e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:49.667377 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.666172 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-workload-certs\") pod \"e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:49.667377 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.666202 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a41dcb03-3288-4425-866e-4473c432b8b5-istio-podinfo\") pod \"e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:49.667377 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.666280 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-credential-socket\") pod \"e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:49.667377 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.666366 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5pz4\" (UniqueName: \"kubernetes.io/projected/a41dcb03-3288-4425-866e-4473c432b8b5-kube-api-access-p5pz4\") pod \"e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:49.667377 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.666400 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-workload-socket\") pod \"e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:49.667377 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.666496 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-istio-envoy\") pod \"e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:49.667377 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.666525 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a41dcb03-3288-4425-866e-4473c432b8b5-istio-token\") pod \"e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:49.667377 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.666557 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-istio-data\") pod \"e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:49.706407 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.706271 5615 generic.go:356] "Generic (PLEG): container finished" podID="bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec" containerID="181c0b2b40fdfbdeca72700c3583e89c33f1f7ea252fa62bffdf92502a2b8092" exitCode=0 Nov 03 21:20:49.708975 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.707459 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-659dffdc8d-xp2zm" Nov 03 21:20:49.720274 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.720235 5615 generic.go:356] "Generic (PLEG): container finished" podID="09aa41ef-008c-45d2-bc4c-e12f58cc83e9" containerID="f2347132c9d50f36b86b17810a6ee2e40388d7e8814fcec260be2b9f9e255f84" exitCode=0 Nov 03 21:20:49.720445 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.720398 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-df45799fd-wjqsf" Nov 03 21:20:49.720680 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.720642 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-659dffdc8d-xp2zm" event={"ID":"bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec","Type":"ContainerDied","Data":"181c0b2b40fdfbdeca72700c3583e89c33f1f7ea252fa62bffdf92502a2b8092"} Nov 03 21:20:49.720824 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.720691 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-659dffdc8d-xp2zm" event={"ID":"bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec","Type":"ContainerDied","Data":"2a8a2c77249b838dddd009be218c97a6b9fea883b21aa03bfae8a7a0b02739df"} Nov 03 21:20:49.720824 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.720709 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-df45799fd-wjqsf" event={"ID":"09aa41ef-008c-45d2-bc4c-e12f58cc83e9","Type":"ContainerDied","Data":"f2347132c9d50f36b86b17810a6ee2e40388d7e8814fcec260be2b9f9e255f84"} Nov 03 21:20:49.720824 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.720726 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-df45799fd-wjqsf" event={"ID":"09aa41ef-008c-45d2-bc4c-e12f58cc83e9","Type":"ContainerDied","Data":"31773de6e585d32903d0cbc100edbbdccf1d08300d80cac9432d8c2db7b87aed"} Nov 03 21:20:49.720824 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.720746 5615 scope.go:122] "RemoveContainer" containerID="181c0b2b40fdfbdeca72700c3583e89c33f1f7ea252fa62bffdf92502a2b8092" Nov 03 21:20:49.736792 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.736766 5615 scope.go:122] "RemoveContainer" containerID="181c0b2b40fdfbdeca72700c3583e89c33f1f7ea252fa62bffdf92502a2b8092" Nov 03 21:20:49.737249 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:20:49.737226 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"181c0b2b40fdfbdeca72700c3583e89c33f1f7ea252fa62bffdf92502a2b8092\": container with ID starting with 181c0b2b40fdfbdeca72700c3583e89c33f1f7ea252fa62bffdf92502a2b8092 not found: ID does not exist" containerID="181c0b2b40fdfbdeca72700c3583e89c33f1f7ea252fa62bffdf92502a2b8092" Nov 03 21:20:49.737331 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.737259 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"181c0b2b40fdfbdeca72700c3583e89c33f1f7ea252fa62bffdf92502a2b8092"} err="failed to get container status \"181c0b2b40fdfbdeca72700c3583e89c33f1f7ea252fa62bffdf92502a2b8092\": rpc error: code = NotFound desc = could not find container \"181c0b2b40fdfbdeca72700c3583e89c33f1f7ea252fa62bffdf92502a2b8092\": container with ID starting with 181c0b2b40fdfbdeca72700c3583e89c33f1f7ea252fa62bffdf92502a2b8092 not found: ID does not exist" Nov 03 21:20:49.737331 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.737277 5615 scope.go:122] "RemoveContainer" containerID="f2347132c9d50f36b86b17810a6ee2e40388d7e8814fcec260be2b9f9e255f84" Nov 03 21:20:49.747188 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.747166 5615 scope.go:122] "RemoveContainer" containerID="f2347132c9d50f36b86b17810a6ee2e40388d7e8814fcec260be2b9f9e255f84" Nov 03 21:20:49.747516 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:20:49.747496 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2347132c9d50f36b86b17810a6ee2e40388d7e8814fcec260be2b9f9e255f84\": container with ID starting with f2347132c9d50f36b86b17810a6ee2e40388d7e8814fcec260be2b9f9e255f84 not found: ID does not exist" containerID="f2347132c9d50f36b86b17810a6ee2e40388d7e8814fcec260be2b9f9e255f84" Nov 03 21:20:49.747569 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.747525 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2347132c9d50f36b86b17810a6ee2e40388d7e8814fcec260be2b9f9e255f84"} err="failed to get container status \"f2347132c9d50f36b86b17810a6ee2e40388d7e8814fcec260be2b9f9e255f84\": rpc error: code = NotFound desc = could not find container \"f2347132c9d50f36b86b17810a6ee2e40388d7e8814fcec260be2b9f9e255f84\": container with ID starting with f2347132c9d50f36b86b17810a6ee2e40388d7e8814fcec260be2b9f9e255f84 not found: ID does not exist" Nov 03 21:20:49.767318 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.767279 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-istio-envoy\") pod \"e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:49.767504 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.767325 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a41dcb03-3288-4425-866e-4473c432b8b5-istio-token\") pod \"e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:49.767504 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.767348 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-istio-data\") pod \"e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:49.767504 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.767408 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a41dcb03-3288-4425-866e-4473c432b8b5-istiod-ca-cert\") pod \"e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:49.767504 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.767441 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-workload-certs\") pod \"e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:49.767504 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.767461 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a41dcb03-3288-4425-866e-4473c432b8b5-istio-podinfo\") pod \"e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:49.767504 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.767503 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-credential-socket\") pod \"e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:49.767830 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.767573 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5pz4\" (UniqueName: \"kubernetes.io/projected/a41dcb03-3288-4425-866e-4473c432b8b5-kube-api-access-p5pz4\") pod \"e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:49.767830 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.767606 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-workload-socket\") pod \"e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:49.767947 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.767886 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-workload-certs\") pod \"e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:49.768011 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.767949 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-workload-socket\") pod \"e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:49.768097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.768055 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-credential-socket\") pod \"e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:49.768281 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.768256 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a41dcb03-3288-4425-866e-4473c432b8b5-istiod-ca-cert\") pod \"e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:49.768364 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.768334 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-istio-data\") pod \"e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:49.769979 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.769942 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-istio-envoy\") pod \"e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:49.770375 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.770352 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a41dcb03-3288-4425-866e-4473c432b8b5-istio-podinfo\") pod \"e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:49.800404 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.800366 5615 patch_prober.go:29] interesting pod/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 21:20:49.800566 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.800437 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" podUID="f285754f-0207-4721-a4f6-62ebfe9a2223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 21:20:49.825915 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.825872 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a41dcb03-3288-4425-866e-4473c432b8b5-istio-token\") pod \"e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:49.826066 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.825990 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5pz4\" (UniqueName: \"kubernetes.io/projected/a41dcb03-3288-4425-866e-4473c432b8b5-kube-api-access-p5pz4\") pod \"e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:49.875203 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.875163 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-659dffdc8d-xp2zm"] Nov 03 21:20:49.879509 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.879480 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2ad2d-659dffdc8d-xp2zm"] Nov 03 21:20:49.998811 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:49.996590 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-df45799fd-wjqsf"] Nov 03 21:20:50.000775 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:50.000746 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2ad2d-df45799fd-wjqsf"] Nov 03 21:20:50.075792 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:50.075760 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:50.498238 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:50.498206 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5"] Nov 03 21:20:50.499672 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:20:50.499644 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda41dcb03_3288_4425_866e_4473c432b8b5.slice/crio-dffadcad6356ef3ac430317a73a9108f7e71ff350ea6988670cfa080b2c3f9eb WatchSource:0}: Error finding container dffadcad6356ef3ac430317a73a9108f7e71ff350ea6988670cfa080b2c3f9eb: Status 404 returned error can't find the container with id dffadcad6356ef3ac430317a73a9108f7e71ff350ea6988670cfa080b2c3f9eb Nov 03 21:20:50.502367 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:50.502332 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:20:50.502455 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:50.502400 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:20:50.502455 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:50.502433 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:20:50.731265 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:50.731229 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" event={"ID":"a41dcb03-3288-4425-866e-4473c432b8b5","Type":"ContainerStarted","Data":"cd8ea78210960b3e4dfab1f2e7e96e9e26e154688aa823643a10a0e440f71893"} Nov 03 21:20:50.731760 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:50.731285 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" event={"ID":"a41dcb03-3288-4425-866e-4473c432b8b5","Type":"ContainerStarted","Data":"dffadcad6356ef3ac430317a73a9108f7e71ff350ea6988670cfa080b2c3f9eb"} Nov 03 21:20:50.733219 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:50.733192 5615 generic.go:356] "Generic (PLEG): container finished" podID="f285754f-0207-4721-a4f6-62ebfe9a2223" containerID="cb2e92e32076d88c076279d331ee8ea131a21a8f0806c0b6eafe04c27e40fdcf" exitCode=0 Nov 03 21:20:50.733341 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:50.733275 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" event={"ID":"f285754f-0207-4721-a4f6-62ebfe9a2223","Type":"ContainerDied","Data":"cb2e92e32076d88c076279d331ee8ea131a21a8f0806c0b6eafe04c27e40fdcf"} Nov 03 21:20:50.733747 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:50.733725 5615 scope.go:122] "RemoveContainer" containerID="cb2e92e32076d88c076279d331ee8ea131a21a8f0806c0b6eafe04c27e40fdcf" Nov 03 21:20:50.735340 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:50.735300 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:20:50.735439 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:50.735387 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:20:50.735506 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:50.735442 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:20:50.848013 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:50.847961 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" podStartSLOduration=1.84794334 podStartE2EDuration="1.84794334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:20:50.846752146 +0000 UTC m=+1945.314530870" watchObservedRunningTime="2026-11-03 21:20:50.84794334 +0000 UTC m=+1945.315722063" Nov 03 21:20:51.077927 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:51.077314 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:51.078124 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:51.078070 5615 patch_prober.go:29] interesting pod/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 03 21:20:51.078212 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:51.078183 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" podUID="a41dcb03-3288-4425-866e-4473c432b8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 03 21:20:51.653211 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:51.653171 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09aa41ef-008c-45d2-bc4c-e12f58cc83e9" path="/var/lib/kubelet/pods/09aa41ef-008c-45d2-bc4c-e12f58cc83e9/volumes" Nov 03 21:20:51.653728 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:51.653707 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec" path="/var/lib/kubelet/pods/bbc4ad39-23d5-4ae5-ab5b-4f42b30de1ec/volumes" Nov 03 21:20:51.742774 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:51.742737 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" event={"ID":"f285754f-0207-4721-a4f6-62ebfe9a2223","Type":"ContainerStarted","Data":"f3307ec28d82ecd57c62ea835bfda015ea8fea567cf815ebe499c9590ccdcb2c"} Nov 03 21:20:52.081252 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:52.081218 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:52.622165 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:52.618472 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:20:52.622165 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:52.618604 5615 patch_prober.go:29] interesting pod/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 21:20:52.622165 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:52.618647 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" podUID="f285754f-0207-4721-a4f6-62ebfe9a2223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 21:20:52.749542 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:52.749500 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:52.750465 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:52.750447 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:20:53.613615 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:53.613575 5615 patch_prober.go:29] interesting pod/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 21:20:53.613918 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:53.613886 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" podUID="f285754f-0207-4721-a4f6-62ebfe9a2223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 21:20:54.620100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:54.617619 5615 patch_prober.go:29] interesting pod/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 21:20:54.620100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:54.617684 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" podUID="f285754f-0207-4721-a4f6-62ebfe9a2223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 21:20:55.621097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:55.614556 5615 patch_prober.go:29] interesting pod/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 21:20:55.621097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:55.614621 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" podUID="f285754f-0207-4721-a4f6-62ebfe9a2223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 21:20:56.622261 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.622220 5615 patch_prober.go:29] interesting pod/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 21:20:56.622904 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.622859 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" podUID="f285754f-0207-4721-a4f6-62ebfe9a2223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 21:20:56.733602 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.733540 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6"] Nov 03 21:20:56.737757 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.737732 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:56.830337 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.830299 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6"] Nov 03 21:20:56.834744 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.834699 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-1190cc-openshift-default-dockercfg-lhw8w\"" Nov 03 21:20:56.856862 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.856831 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istio-podinfo\") pod \"e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:56.856862 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.856868 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istio-data\") pod \"e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:56.857159 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.856888 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzrrk\" (UniqueName: \"kubernetes.io/projected/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-kube-api-access-vzrrk\") pod \"e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:56.857159 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.857008 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-credential-socket\") pod \"e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:56.857159 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.857038 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istiod-ca-cert\") pod \"e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:56.857311 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.857214 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istio-token\") pod \"e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:56.857311 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.857272 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istio-envoy\") pod \"e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:56.857389 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.857374 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-workload-certs\") pod \"e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:56.857506 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.857485 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-workload-socket\") pod \"e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:56.958962 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.958868 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istio-podinfo\") pod \"e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:56.958962 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.958926 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istio-data\") pod \"e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:56.958962 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.958960 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzrrk\" (UniqueName: \"kubernetes.io/projected/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-kube-api-access-vzrrk\") pod \"e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:56.959319 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.959029 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-credential-socket\") pod \"e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:56.959319 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.959067 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istiod-ca-cert\") pod \"e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:56.959319 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.959207 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istio-token\") pod \"e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:56.959319 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.959257 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istio-envoy\") pod \"e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:56.959534 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.959330 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-workload-certs\") pod \"e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:56.959534 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.959439 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-workload-socket\") pod \"e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:56.959534 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.959467 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istio-data\") pod \"e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:56.959534 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.959494 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-credential-socket\") pod \"e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:56.959776 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.959753 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-workload-certs\") pod \"e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:56.959898 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.959869 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-workload-socket\") pod \"e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:56.959980 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.959948 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istiod-ca-cert\") pod \"e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:56.961604 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.961563 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istio-podinfo\") pod \"e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:56.961725 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:56.961651 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istio-envoy\") pod \"e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:57.055298 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:57.054755 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istio-token\") pod \"e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:57.055298 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:57.054894 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzrrk\" (UniqueName: \"kubernetes.io/projected/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-kube-api-access-vzrrk\") pod \"e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:57.055298 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:57.055037 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:57.599767 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:57.599736 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6"] Nov 03 21:20:57.605880 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:57.605842 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:20:57.605973 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:57.605914 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:20:57.605973 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:57.605944 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:20:57.617859 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:57.617831 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:20:57.618329 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:57.618303 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:20:57.619289 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:57.619251 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:20:57.784178 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:57.784136 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" event={"ID":"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1","Type":"ContainerStarted","Data":"e64f9e2af5fb776cb21f76dde321e501ed0e0dc217d103ad9bb140da94cf4de9"} Nov 03 21:20:57.784755 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:57.784204 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" event={"ID":"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1","Type":"ContainerStarted","Data":"f2e64e91ca19a27af0bb0a35599e9930a2c9ef42e3df7b1fc4598812d1fdab44"} Nov 03 21:20:58.020370 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:58.020311 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" podStartSLOduration=2.020295591 podStartE2EDuration="2.020295591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:20:58.018251581 +0000 UTC m=+1952.486030305" watchObservedRunningTime="2026-11-03 21:20:58.020295591 +0000 UTC m=+1952.488074313" Nov 03 21:20:58.055954 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:58.055915 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:58.057659 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:58.057635 5615 patch_prober.go:29] interesting pod/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 21:20:58.057786 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:58.057725 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" podUID="d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 21:20:59.061006 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:59.060975 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:59.820850 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:59.820815 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:20:59.821892 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:20:59.821869 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:21:10.619815 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:21:10.619769 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:21:24.727392 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:24.727356 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a77060-58c7974dd-mk4wt"] Nov 03 21:21:24.835537 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:24.835501 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk"] Nov 03 21:21:26.486099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:26.485506 5615 scope.go:122] "RemoveContainer" containerID="f2cc3d4302d9e7cfa7609348651f78e81005812a5d1c3f34dc1f9c744ec2ee24" Nov 03 21:21:28.423098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:28.422708 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-f6d7b55fb-k5rwz"] Nov 03 21:21:28.432097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:28.430163 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-f6d7b55fb-k5rwz" Nov 03 21:21:28.485099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:28.483183 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-f6d7b55fb-k5rwz"] Nov 03 21:21:28.493926 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:28.493891 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee45aa1b-3c2e-4721-bf25-28631bcb0110-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-812ef9-f6d7b55fb-k5rwz\" (UID: \"ee45aa1b-3c2e-4721-bf25-28631bcb0110\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-f6d7b55fb-k5rwz" Nov 03 21:21:28.494149 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:28.494086 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf8hq\" (UniqueName: \"kubernetes.io/projected/ee45aa1b-3c2e-4721-bf25-28631bcb0110-kube-api-access-cf8hq\") pod \"payload-pre-processing-e2e-ipp-b-812ef9-f6d7b55fb-k5rwz\" (UID: \"ee45aa1b-3c2e-4721-bf25-28631bcb0110\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-f6d7b55fb-k5rwz" Nov 03 21:21:28.521101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:28.518625 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-812ef9\"" Nov 03 21:21:28.521101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:28.518845 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-812ef9-dockercfg-tvxcf\"" Nov 03 21:21:28.594988 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:28.594944 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee45aa1b-3c2e-4721-bf25-28631bcb0110-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-812ef9-f6d7b55fb-k5rwz\" (UID: \"ee45aa1b-3c2e-4721-bf25-28631bcb0110\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-f6d7b55fb-k5rwz" Nov 03 21:21:28.595234 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:28.595114 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf8hq\" (UniqueName: \"kubernetes.io/projected/ee45aa1b-3c2e-4721-bf25-28631bcb0110-kube-api-access-cf8hq\") pod \"payload-pre-processing-e2e-ipp-b-812ef9-f6d7b55fb-k5rwz\" (UID: \"ee45aa1b-3c2e-4721-bf25-28631bcb0110\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-f6d7b55fb-k5rwz" Nov 03 21:21:28.596051 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:28.596029 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee45aa1b-3c2e-4721-bf25-28631bcb0110-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-812ef9-f6d7b55fb-k5rwz\" (UID: \"ee45aa1b-3c2e-4721-bf25-28631bcb0110\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-f6d7b55fb-k5rwz" Nov 03 21:21:28.673334 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:28.672924 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf8hq\" (UniqueName: \"kubernetes.io/projected/ee45aa1b-3c2e-4721-bf25-28631bcb0110-kube-api-access-cf8hq\") pod \"payload-pre-processing-e2e-ipp-b-812ef9-f6d7b55fb-k5rwz\" (UID: \"ee45aa1b-3c2e-4721-bf25-28631bcb0110\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-f6d7b55fb-k5rwz" Nov 03 21:21:28.789324 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:28.789287 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-f6d7b55fb-k5rwz" Nov 03 21:21:29.328291 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:29.314987 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-f6d7b55fb-k5rwz"] Nov 03 21:21:29.728684 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:29.728592 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-58c7974dd-mk4wt" podUID="26954941-5e89-4477-b16b-f59abeaf006a" containerName="payload-processing" containerID="cri-o://5c1bb921cbb523c4b51511c287b24860f097c8b2554e77d0bdf823866d8468a1" gracePeriod=25 Nov 03 21:21:29.836426 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:29.836385 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk" podUID="bbc36145-9288-4ffe-9c5b-d836f824fd5f" containerName="payload-pre-processing" containerID="cri-o://e968b139d78bcd38e4f51051e3cc5bd43961d678d53e2f86f86e9441ad12b9d9" gracePeriod=25 Nov 03 21:21:29.886958 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:21:29.886914 5615 logging.go:55] [core] [Channel #2963 SubChannel #2964]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.118:50052", ServerName: "10.134.0.118:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.118:50052: connect: connection refused" Nov 03 21:21:30.036189 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:30.036158 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-58c7974dd-mk4wt" Nov 03 21:21:30.120145 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:30.120104 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-f6d7b55fb-k5rwz" event={"ID":"ee45aa1b-3c2e-4721-bf25-28631bcb0110","Type":"ContainerStarted","Data":"350b82da0033d2a2a7e9876da1ee7ddc46057edf41807772bd0a63b809193b07"} Nov 03 21:21:30.120435 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:30.120417 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-f6d7b55fb-k5rwz" event={"ID":"ee45aa1b-3c2e-4721-bf25-28631bcb0110","Type":"ContainerStarted","Data":"fd49299d2723506d1898a0b8b8c58588132d4f17c796a8d81ee4b7a32b75fd2b"} Nov 03 21:21:30.121450 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:30.121426 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-f6d7b55fb-k5rwz" Nov 03 21:21:30.124394 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:30.124368 5615 generic.go:356] "Generic (PLEG): container finished" podID="bbc36145-9288-4ffe-9c5b-d836f824fd5f" containerID="e968b139d78bcd38e4f51051e3cc5bd43961d678d53e2f86f86e9441ad12b9d9" exitCode=0 Nov 03 21:21:30.125008 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:30.124909 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk" event={"ID":"bbc36145-9288-4ffe-9c5b-d836f824fd5f","Type":"ContainerDied","Data":"e968b139d78bcd38e4f51051e3cc5bd43961d678d53e2f86f86e9441ad12b9d9"} Nov 03 21:21:30.141001 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:30.135596 5615 generic.go:356] "Generic (PLEG): container finished" podID="26954941-5e89-4477-b16b-f59abeaf006a" containerID="5c1bb921cbb523c4b51511c287b24860f097c8b2554e77d0bdf823866d8468a1" exitCode=0 Nov 03 21:21:30.141001 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:30.135659 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-58c7974dd-mk4wt" event={"ID":"26954941-5e89-4477-b16b-f59abeaf006a","Type":"ContainerDied","Data":"5c1bb921cbb523c4b51511c287b24860f097c8b2554e77d0bdf823866d8468a1"} Nov 03 21:21:30.141001 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:30.135712 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-58c7974dd-mk4wt" event={"ID":"26954941-5e89-4477-b16b-f59abeaf006a","Type":"ContainerDied","Data":"d3dd845b446b636e22a59451b8d0dd0576df228881effe2c98c8342a33d0bdad"} Nov 03 21:21:30.141001 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:30.135740 5615 scope.go:122] "RemoveContainer" containerID="5c1bb921cbb523c4b51511c287b24860f097c8b2554e77d0bdf823866d8468a1" Nov 03 21:21:30.141001 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:30.136118 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-58c7974dd-mk4wt" Nov 03 21:21:30.183517 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:30.177180 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26954941-5e89-4477-b16b-f59abeaf006a-kube-api-access-5vzxp\" (UniqueName: \"kubernetes.io/projected/26954941-5e89-4477-b16b-f59abeaf006a-kube-api-access-5vzxp\") pod \"26954941-5e89-4477-b16b-f59abeaf006a\" (UID: \"26954941-5e89-4477-b16b-f59abeaf006a\") " Nov 03 21:21:30.183517 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:30.177339 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26954941-5e89-4477-b16b-f59abeaf006a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26954941-5e89-4477-b16b-f59abeaf006a-plugins-config-volume\") pod \"26954941-5e89-4477-b16b-f59abeaf006a\" (UID: \"26954941-5e89-4477-b16b-f59abeaf006a\") " Nov 03 21:21:30.183517 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:30.179275 5615 scope.go:122] "RemoveContainer" containerID="5c1bb921cbb523c4b51511c287b24860f097c8b2554e77d0bdf823866d8468a1" Nov 03 21:21:30.183517 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:30.180133 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26954941-5e89-4477-b16b-f59abeaf006a-plugins-config-volume" pod "26954941-5e89-4477-b16b-f59abeaf006a" (UID: "26954941-5e89-4477-b16b-f59abeaf006a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:21:30.183517 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:30.181449 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26954941-5e89-4477-b16b-f59abeaf006a-kube-api-access-5vzxp" pod "26954941-5e89-4477-b16b-f59abeaf006a" (UID: "26954941-5e89-4477-b16b-f59abeaf006a"). InnerVolumeSpecName "kube-api-access-5vzxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:21:30.184551 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:21:30.184429 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c1bb921cbb523c4b51511c287b24860f097c8b2554e77d0bdf823866d8468a1\": container with ID starting with 5c1bb921cbb523c4b51511c287b24860f097c8b2554e77d0bdf823866d8468a1 not found: ID does not exist" containerID="5c1bb921cbb523c4b51511c287b24860f097c8b2554e77d0bdf823866d8468a1" Nov 03 21:21:30.184551 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:30.184474 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c1bb921cbb523c4b51511c287b24860f097c8b2554e77d0bdf823866d8468a1"} err="failed to get container status \"5c1bb921cbb523c4b51511c287b24860f097c8b2554e77d0bdf823866d8468a1\": rpc error: code = NotFound desc = could not find container \"5c1bb921cbb523c4b51511c287b24860f097c8b2554e77d0bdf823866d8468a1\": container with ID starting with 5c1bb921cbb523c4b51511c287b24860f097c8b2554e77d0bdf823866d8468a1 not found: ID does not exist" Nov 03 21:21:30.206901 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:30.206776 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk" Nov 03 21:21:30.286102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:30.285881 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbc36145-9288-4ffe-9c5b-d836f824fd5f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc36145-9288-4ffe-9c5b-d836f824fd5f-plugins-config-volume\") pod \"bbc36145-9288-4ffe-9c5b-d836f824fd5f\" (UID: \"bbc36145-9288-4ffe-9c5b-d836f824fd5f\") " Nov 03 21:21:30.286532 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:30.286502 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbc36145-9288-4ffe-9c5b-d836f824fd5f-kube-api-access-fsxcw\" (UniqueName: \"kubernetes.io/projected/bbc36145-9288-4ffe-9c5b-d836f824fd5f-kube-api-access-fsxcw\") pod \"bbc36145-9288-4ffe-9c5b-d836f824fd5f\" (UID: \"bbc36145-9288-4ffe-9c5b-d836f824fd5f\") " Nov 03 21:21:30.291160 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:30.287293 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vzxp\" (UniqueName: \"kubernetes.io/projected/26954941-5e89-4477-b16b-f59abeaf006a-kube-api-access-5vzxp\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:21:30.291160 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:30.287324 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26954941-5e89-4477-b16b-f59abeaf006a-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:21:30.291160 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:30.288063 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc36145-9288-4ffe-9c5b-d836f824fd5f-plugins-config-volume" pod "bbc36145-9288-4ffe-9c5b-d836f824fd5f" (UID: "bbc36145-9288-4ffe-9c5b-d836f824fd5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:21:30.312864 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:30.312777 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc36145-9288-4ffe-9c5b-d836f824fd5f-kube-api-access-fsxcw" pod "bbc36145-9288-4ffe-9c5b-d836f824fd5f" (UID: "bbc36145-9288-4ffe-9c5b-d836f824fd5f"). InnerVolumeSpecName "kube-api-access-fsxcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:21:30.388130 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:30.388049 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc36145-9288-4ffe-9c5b-d836f824fd5f-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:21:30.388130 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:30.388124 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsxcw\" (UniqueName: \"kubernetes.io/projected/bbc36145-9288-4ffe-9c5b-d836f824fd5f-kube-api-access-fsxcw\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:21:30.429751 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:30.429680 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-f6d7b55fb-k5rwz" podStartSLOduration=2.429659442 podStartE2EDuration="2.429659442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:21:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:21:30.313220015 +0000 UTC m=+1984.780998743" watchObservedRunningTime="2026-11-03 21:21:30.429659442 +0000 UTC m=+1984.897438165" Nov 03 21:21:30.617973 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:30.617877 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a77060-58c7974dd-mk4wt"] Nov 03 21:21:30.621525 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:30.621492 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a77060-58c7974dd-mk4wt"] Nov 03 21:21:30.886937 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:30.886836 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk" podUID="bbc36145-9288-4ffe-9c5b-d836f824fd5f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.118:50052\" within 1s: context deadline exceeded" Nov 03 21:21:31.144800 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:31.144711 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk" Nov 03 21:21:31.144800 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:31.144741 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk" event={"ID":"bbc36145-9288-4ffe-9c5b-d836f824fd5f","Type":"ContainerDied","Data":"558019d27bcd3f7a103065c223bb7bc631aadd1bf55024b0f753226f1176456a"} Nov 03 21:21:31.145043 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:31.144801 5615 scope.go:122] "RemoveContainer" containerID="e968b139d78bcd38e4f51051e3cc5bd43961d678d53e2f86f86e9441ad12b9d9" Nov 03 21:21:31.310024 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:31.309990 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk"] Nov 03 21:21:31.312580 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:31.312557 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-dfc7f887f-w5zhk"] Nov 03 21:21:31.653163 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:31.653132 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26954941-5e89-4477-b16b-f59abeaf006a" path="/var/lib/kubelet/pods/26954941-5e89-4477-b16b-f59abeaf006a/volumes" Nov 03 21:21:31.653676 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:31.653661 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbc36145-9288-4ffe-9c5b-d836f824fd5f" path="/var/lib/kubelet/pods/bbc36145-9288-4ffe-9c5b-d836f824fd5f/volumes" Nov 03 21:21:40.620724 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:21:40.620640 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:21:42.156495 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:42.156461 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-f6d7b55fb-k5rwz" Nov 03 21:21:54.735913 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:54.735876 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t"] Nov 03 21:21:54.736517 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:54.736502 5615 cpu_manager.go:414] "containerMap: removing container" podUID="26954941-5e89-4477-b16b-f59abeaf006a" containerName="payload-processing" Nov 03 21:21:54.736572 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:54.736518 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26954941-5e89-4477-b16b-f59abeaf006a" containerName="payload-processing" Nov 03 21:21:54.736572 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:54.736562 5615 cpu_manager.go:414] "containerMap: removing container" podUID="bbc36145-9288-4ffe-9c5b-d836f824fd5f" containerName="payload-pre-processing" Nov 03 21:21:54.736572 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:54.736568 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbc36145-9288-4ffe-9c5b-d836f824fd5f" containerName="payload-pre-processing" Nov 03 21:21:54.736682 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:54.736658 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbc36145-9288-4ffe-9c5b-d836f824fd5f" containerName="payload-pre-processing" Nov 03 21:21:54.736682 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:54.736670 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="26954941-5e89-4477-b16b-f59abeaf006a" containerName="payload-processing" Nov 03 21:21:54.740068 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:54.740047 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t" Nov 03 21:21:54.839331 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:54.839289 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-7af9dd-serving-cert\"" Nov 03 21:21:54.842022 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:54.841993 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t"] Nov 03 21:21:54.922956 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:54.922897 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/314a2820-1a0c-477b-adab-a2428c552b65-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t\" (UID: \"314a2820-1a0c-477b-adab-a2428c552b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t" Nov 03 21:21:54.923201 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:54.923169 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6rs6\" (UniqueName: \"kubernetes.io/projected/314a2820-1a0c-477b-adab-a2428c552b65-kube-api-access-r6rs6\") pod \"maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t\" (UID: \"314a2820-1a0c-477b-adab-a2428c552b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t" Nov 03 21:21:54.923287 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:54.923253 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/314a2820-1a0c-477b-adab-a2428c552b65-metrics-certs\") pod \"maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t\" (UID: \"314a2820-1a0c-477b-adab-a2428c552b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t" Nov 03 21:21:54.923349 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:54.923299 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/314a2820-1a0c-477b-adab-a2428c552b65-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t\" (UID: \"314a2820-1a0c-477b-adab-a2428c552b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t" Nov 03 21:21:55.051407 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:55.051307 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6rs6\" (UniqueName: \"kubernetes.io/projected/314a2820-1a0c-477b-adab-a2428c552b65-kube-api-access-r6rs6\") pod \"maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t\" (UID: \"314a2820-1a0c-477b-adab-a2428c552b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t" Nov 03 21:21:55.051735 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:55.051705 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/314a2820-1a0c-477b-adab-a2428c552b65-metrics-certs\") pod \"maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t\" (UID: \"314a2820-1a0c-477b-adab-a2428c552b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t" Nov 03 21:21:55.051890 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:55.051874 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/314a2820-1a0c-477b-adab-a2428c552b65-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t\" (UID: \"314a2820-1a0c-477b-adab-a2428c552b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t" Nov 03 21:21:55.052126 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:55.052106 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/314a2820-1a0c-477b-adab-a2428c552b65-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t\" (UID: \"314a2820-1a0c-477b-adab-a2428c552b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t" Nov 03 21:21:55.061246 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:55.061205 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/314a2820-1a0c-477b-adab-a2428c552b65-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t\" (UID: \"314a2820-1a0c-477b-adab-a2428c552b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t" Nov 03 21:21:55.087124 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:55.085801 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/314a2820-1a0c-477b-adab-a2428c552b65-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t\" (UID: \"314a2820-1a0c-477b-adab-a2428c552b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t" Nov 03 21:21:55.087124 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:55.086389 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/314a2820-1a0c-477b-adab-a2428c552b65-metrics-certs\") pod \"maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t\" (UID: \"314a2820-1a0c-477b-adab-a2428c552b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t" Nov 03 21:21:55.118362 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:55.118327 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6rs6\" (UniqueName: \"kubernetes.io/projected/314a2820-1a0c-477b-adab-a2428c552b65-kube-api-access-r6rs6\") pod \"maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t\" (UID: \"314a2820-1a0c-477b-adab-a2428c552b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t" Nov 03 21:21:55.353956 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:55.353862 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t" Nov 03 21:21:55.728386 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:55.728354 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t"] Nov 03 21:21:55.729723 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:21:55.729691 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod314a2820_1a0c_477b_adab_a2428c552b65.slice/crio-01de04916ae7f9fec618980f85689f89644f8f054de128b3c1c8d2629d0689e8 WatchSource:0}: Error finding container 01de04916ae7f9fec618980f85689f89644f8f054de128b3c1c8d2629d0689e8: Status 404 returned error can't find the container with id 01de04916ae7f9fec618980f85689f89644f8f054de128b3c1c8d2629d0689e8 Nov 03 21:21:56.307863 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:56.307820 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t" event={"ID":"314a2820-1a0c-477b-adab-a2428c552b65","Type":"ContainerStarted","Data":"01de04916ae7f9fec618980f85689f89644f8f054de128b3c1c8d2629d0689e8"} Nov 03 21:21:57.319136 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:57.319064 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t" event={"ID":"314a2820-1a0c-477b-adab-a2428c552b65","Type":"ContainerStarted","Data":"47a0fd4dadc6a6146affe1a6ca85ea9f527a7b33b3ab46ad1b640d812876deb8"} Nov 03 21:21:57.319665 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:57.319448 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t" Nov 03 21:21:57.434581 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:21:57.434522 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t" podStartSLOduration=2.470608846 podStartE2EDuration="3.434506671s" totalImagesPullingTime="963.897825ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:21:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:21:57.431571123 +0000 UTC m=+2011.899349847" watchObservedRunningTime="2026-11-03 21:21:57.434506671 +0000 UTC m=+2011.902285393" Nov 03 21:22:03.331011 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:03.330978 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t" Nov 03 21:22:10.619521 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:22:10.619479 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:22:23.146968 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:23.146434 5615 patch_prober.go:29] interesting pod/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": read tcp 10.134.0.2:51056->10.134.0.129:15021: read: connection reset by peer" start-of-body= Nov 03 21:22:23.146968 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:23.146501 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" podUID="a41dcb03-3288-4425-866e-4473c432b8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": read tcp 10.134.0.2:51056->10.134.0.129:15021: read: connection reset by peer" Nov 03 21:22:23.567488 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:23.567451 5615 generic.go:356] "Generic (PLEG): container finished" podID="a41dcb03-3288-4425-866e-4473c432b8b5" containerID="cd8ea78210960b3e4dfab1f2e7e96e9e26e154688aa823643a10a0e440f71893" exitCode=0 Nov 03 21:22:23.567723 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:23.567533 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" event={"ID":"a41dcb03-3288-4425-866e-4473c432b8b5","Type":"ContainerDied","Data":"cd8ea78210960b3e4dfab1f2e7e96e9e26e154688aa823643a10a0e440f71893"} Nov 03 21:22:23.568043 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:23.568024 5615 scope.go:122] "RemoveContainer" containerID="cd8ea78210960b3e4dfab1f2e7e96e9e26e154688aa823643a10a0e440f71893" Nov 03 21:22:23.569648 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:23.569613 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:22:23.569762 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:23.569720 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:22:23.569829 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:23.569773 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:22:24.575312 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:24.575274 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" event={"ID":"a41dcb03-3288-4425-866e-4473c432b8b5","Type":"ContainerStarted","Data":"6af3dcef8ada1fd8d46dfcc4c69138b7080a8889d1fa1e5ad988ac2efdf5748f"} Nov 03 21:22:25.076369 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:25.076334 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:22:25.076554 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:25.076525 5615 patch_prober.go:29] interesting pod/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 03 21:22:25.076609 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:25.076569 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" podUID="a41dcb03-3288-4425-866e-4473c432b8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 03 21:22:26.076210 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:26.076170 5615 patch_prober.go:29] interesting pod/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 03 21:22:26.076579 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:26.076234 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" podUID="a41dcb03-3288-4425-866e-4473c432b8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 03 21:22:27.076186 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:27.076142 5615 patch_prober.go:29] interesting pod/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 03 21:22:27.076406 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:27.076209 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" podUID="a41dcb03-3288-4425-866e-4473c432b8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 03 21:22:28.076211 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:28.076166 5615 patch_prober.go:29] interesting pod/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 03 21:22:28.076437 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:28.076317 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" podUID="a41dcb03-3288-4425-866e-4473c432b8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 03 21:22:29.076247 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:29.076205 5615 patch_prober.go:29] interesting pod/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 03 21:22:29.076438 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:29.076274 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" podUID="a41dcb03-3288-4425-866e-4473c432b8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 03 21:22:29.157233 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:29.157193 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj"] Nov 03 21:22:29.161821 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:29.161791 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj" Nov 03 21:22:29.220100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:29.214691 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a77060\"" Nov 03 21:22:29.230301 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:29.230273 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a77060-dockercfg-886p8\"" Nov 03 21:22:29.264108 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:29.264050 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq5md\" (UniqueName: \"kubernetes.io/projected/cd915ca2-fb6f-4a5b-bb22-f281df8f7337-kube-api-access-qq5md\") pod \"payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj\" (UID: \"cd915ca2-fb6f-4a5b-bb22-f281df8f7337\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj" Nov 03 21:22:29.264545 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:29.264517 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd915ca2-fb6f-4a5b-bb22-f281df8f7337-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj\" (UID: \"cd915ca2-fb6f-4a5b-bb22-f281df8f7337\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj" Nov 03 21:22:29.265300 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:29.265271 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj"] Nov 03 21:22:29.366547 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:29.366453 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq5md\" (UniqueName: \"kubernetes.io/projected/cd915ca2-fb6f-4a5b-bb22-f281df8f7337-kube-api-access-qq5md\") pod \"payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj\" (UID: \"cd915ca2-fb6f-4a5b-bb22-f281df8f7337\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj" Nov 03 21:22:29.366736 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:29.366591 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd915ca2-fb6f-4a5b-bb22-f281df8f7337-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj\" (UID: \"cd915ca2-fb6f-4a5b-bb22-f281df8f7337\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj" Nov 03 21:22:29.367314 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:29.367292 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd915ca2-fb6f-4a5b-bb22-f281df8f7337-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj\" (UID: \"cd915ca2-fb6f-4a5b-bb22-f281df8f7337\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj" Nov 03 21:22:29.431943 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:29.431903 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq5md\" (UniqueName: \"kubernetes.io/projected/cd915ca2-fb6f-4a5b-bb22-f281df8f7337-kube-api-access-qq5md\") pod \"payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj\" (UID: \"cd915ca2-fb6f-4a5b-bb22-f281df8f7337\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj" Nov 03 21:22:29.477267 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:29.477232 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj" Nov 03 21:22:29.892202 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:29.892019 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj"] Nov 03 21:22:29.895979 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:22:29.895944 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd915ca2_fb6f_4a5b_bb22_f281df8f7337.slice/crio-556cb1f34ed37668579ffd0e355f92124b38dbc700d733e9c7ab7fdc646be2cb WatchSource:0}: Error finding container 556cb1f34ed37668579ffd0e355f92124b38dbc700d733e9c7ab7fdc646be2cb: Status 404 returned error can't find the container with id 556cb1f34ed37668579ffd0e355f92124b38dbc700d733e9c7ab7fdc646be2cb Nov 03 21:22:30.084438 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:30.084403 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:22:30.085351 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:30.085323 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:22:30.086487 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:30.086465 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:22:30.654790 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:30.654758 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj" event={"ID":"cd915ca2-fb6f-4a5b-bb22-f281df8f7337","Type":"ContainerStarted","Data":"bfcedbe952a40e740fb7316c71c1c1e0c8cbb5081966afc514e1a268f9e2090e"} Nov 03 21:22:30.655317 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:30.655290 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj" Nov 03 21:22:30.655465 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:30.655334 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj" event={"ID":"cd915ca2-fb6f-4a5b-bb22-f281df8f7337","Type":"ContainerStarted","Data":"556cb1f34ed37668579ffd0e355f92124b38dbc700d733e9c7ab7fdc646be2cb"} Nov 03 21:22:40.619017 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:22:40.618977 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:22:41.661977 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:41.661945 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj" Nov 03 21:22:41.775298 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:41.775238 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj" podStartSLOduration=12.775224061 podStartE2EDuration="12.775224061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:22:30.777522981 +0000 UTC m=+2045.245301962" watchObservedRunningTime="2026-11-03 21:22:41.775224061 +0000 UTC m=+2056.243002774" Nov 03 21:22:46.589590 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:46.589558 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-r5zp7_40e8212c-49a4-4a50-add6-0ee8a55429b2/payload-processing/0.log" Nov 03 21:22:46.833448 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:46.833409 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-r5zp7_40e8212c-49a4-4a50-add6-0ee8a55429b2/payload-processing/0.log" Nov 03 21:22:47.071493 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:47.071459 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj_cd915ca2-fb6f-4a5b-bb22-f281df8f7337/payload-processing/0.log" Nov 03 21:22:52.806024 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:52.805991 5615 generic.go:356] "Generic (PLEG): container finished" podID="d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1" containerID="e64f9e2af5fb776cb21f76dde321e501ed0e0dc217d103ad9bb140da94cf4de9" exitCode=0 Nov 03 21:22:52.806488 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:52.806068 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" event={"ID":"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1","Type":"ContainerDied","Data":"e64f9e2af5fb776cb21f76dde321e501ed0e0dc217d103ad9bb140da94cf4de9"} Nov 03 21:22:52.806531 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:52.806522 5615 scope.go:122] "RemoveContainer" containerID="e64f9e2af5fb776cb21f76dde321e501ed0e0dc217d103ad9bb140da94cf4de9" Nov 03 21:22:52.808159 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:52.808131 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:22:52.808235 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:52.808202 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:22:52.808272 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:52.808233 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:22:53.827847 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:53.827810 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" event={"ID":"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1","Type":"ContainerStarted","Data":"d50cce2819f0cc2a8116d79740ccd9cf8b8734cd5fc015afa8ece7d9883683ee"} Nov 03 21:22:54.056211 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:54.056170 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:22:54.056425 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:54.056359 5615 patch_prober.go:29] interesting pod/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 21:22:54.056425 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:54.056411 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" podUID="d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 21:22:55.056449 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:55.056410 5615 patch_prober.go:29] interesting pod/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 21:22:55.056851 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:55.056479 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" podUID="d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 21:22:55.957028 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:55.956985 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt"] Nov 03 21:22:55.957300 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:55.957266 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt" podUID="dfacf7de-fac7-47ee-84b7-4fe5a51e0b26" containerName="maas-api" containerID="cri-o://da0724ab87045a3202c7b2267d7e0a78c187939893f8ed35ecceff2fa831f41a" gracePeriod=30 Nov 03 21:22:56.056650 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.056610 5615 patch_prober.go:29] interesting pod/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 21:22:56.057061 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.056681 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" podUID="d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 21:22:56.396142 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.391515 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-66513c-7765fcd666-6l4wb"] Nov 03 21:22:56.396142 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.391742 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-7765fcd666-6l4wb" podUID="911bcfcb-bd59-4477-8b70-02f725d43ff7" containerName="payload-processing" containerID="cri-o://4c28ce2c3750835cd5e64e3f699c56a8f63b8812369a2a319683c28949b8200e" gracePeriod=30 Nov 03 21:22:56.465701 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.465665 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt" Nov 03 21:22:56.561236 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.561201 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfacf7de-fac7-47ee-84b7-4fe5a51e0b26-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfacf7de-fac7-47ee-84b7-4fe5a51e0b26-metrics-certs\") pod \"dfacf7de-fac7-47ee-84b7-4fe5a51e0b26\" (UID: \"dfacf7de-fac7-47ee-84b7-4fe5a51e0b26\") " Nov 03 21:22:56.561408 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.561366 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfacf7de-fac7-47ee-84b7-4fe5a51e0b26-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfacf7de-fac7-47ee-84b7-4fe5a51e0b26-openshift-service-ca\") pod \"dfacf7de-fac7-47ee-84b7-4fe5a51e0b26\" (UID: \"dfacf7de-fac7-47ee-84b7-4fe5a51e0b26\") " Nov 03 21:22:56.561496 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.561407 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfacf7de-fac7-47ee-84b7-4fe5a51e0b26-kube-api-access-ll552\" (UniqueName: \"kubernetes.io/projected/dfacf7de-fac7-47ee-84b7-4fe5a51e0b26-kube-api-access-ll552\") pod \"dfacf7de-fac7-47ee-84b7-4fe5a51e0b26\" (UID: \"dfacf7de-fac7-47ee-84b7-4fe5a51e0b26\") " Nov 03 21:22:56.561496 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.561484 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfacf7de-fac7-47ee-84b7-4fe5a51e0b26-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfacf7de-fac7-47ee-84b7-4fe5a51e0b26-maas-api-tls\") pod \"dfacf7de-fac7-47ee-84b7-4fe5a51e0b26\" (UID: \"dfacf7de-fac7-47ee-84b7-4fe5a51e0b26\") " Nov 03 21:22:56.561772 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.561744 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfacf7de-fac7-47ee-84b7-4fe5a51e0b26-openshift-service-ca" pod "dfacf7de-fac7-47ee-84b7-4fe5a51e0b26" (UID: "dfacf7de-fac7-47ee-84b7-4fe5a51e0b26"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:22:56.561988 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.561969 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfacf7de-fac7-47ee-84b7-4fe5a51e0b26-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:56.563707 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.563613 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfacf7de-fac7-47ee-84b7-4fe5a51e0b26-metrics-certs" pod "dfacf7de-fac7-47ee-84b7-4fe5a51e0b26" (UID: "dfacf7de-fac7-47ee-84b7-4fe5a51e0b26"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:22:56.563707 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.563671 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfacf7de-fac7-47ee-84b7-4fe5a51e0b26-maas-api-tls" pod "dfacf7de-fac7-47ee-84b7-4fe5a51e0b26" (UID: "dfacf7de-fac7-47ee-84b7-4fe5a51e0b26"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:22:56.564328 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.564307 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfacf7de-fac7-47ee-84b7-4fe5a51e0b26-kube-api-access-ll552" pod "dfacf7de-fac7-47ee-84b7-4fe5a51e0b26" (UID: "dfacf7de-fac7-47ee-84b7-4fe5a51e0b26"). InnerVolumeSpecName "kube-api-access-ll552". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:22:56.663431 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.663388 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll552\" (UniqueName: \"kubernetes.io/projected/dfacf7de-fac7-47ee-84b7-4fe5a51e0b26-kube-api-access-ll552\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:56.663599 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.663435 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfacf7de-fac7-47ee-84b7-4fe5a51e0b26-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:56.663599 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.663450 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfacf7de-fac7-47ee-84b7-4fe5a51e0b26-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:56.775149 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.775120 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-7765fcd666-6l4wb" Nov 03 21:22:56.849211 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.848968 5615 generic.go:356] "Generic (PLEG): container finished" podID="911bcfcb-bd59-4477-8b70-02f725d43ff7" containerID="4c28ce2c3750835cd5e64e3f699c56a8f63b8812369a2a319683c28949b8200e" exitCode=0 Nov 03 21:22:56.849211 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.849014 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-7765fcd666-6l4wb" Nov 03 21:22:56.849211 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.849051 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-7765fcd666-6l4wb" event={"ID":"911bcfcb-bd59-4477-8b70-02f725d43ff7","Type":"ContainerDied","Data":"4c28ce2c3750835cd5e64e3f699c56a8f63b8812369a2a319683c28949b8200e"} Nov 03 21:22:56.849211 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.849120 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-7765fcd666-6l4wb" event={"ID":"911bcfcb-bd59-4477-8b70-02f725d43ff7","Type":"ContainerDied","Data":"ac5c76af7c14c3822c26d5d8a9f976d329d11e7335b92adbe983a59ca0a68080"} Nov 03 21:22:56.849211 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.849147 5615 scope.go:122] "RemoveContainer" containerID="4c28ce2c3750835cd5e64e3f699c56a8f63b8812369a2a319683c28949b8200e" Nov 03 21:22:56.851017 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.850994 5615 generic.go:356] "Generic (PLEG): container finished" podID="dfacf7de-fac7-47ee-84b7-4fe5a51e0b26" containerID="da0724ab87045a3202c7b2267d7e0a78c187939893f8ed35ecceff2fa831f41a" exitCode=0 Nov 03 21:22:56.851141 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.851051 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt" event={"ID":"dfacf7de-fac7-47ee-84b7-4fe5a51e0b26","Type":"ContainerDied","Data":"da0724ab87045a3202c7b2267d7e0a78c187939893f8ed35ecceff2fa831f41a"} Nov 03 21:22:56.851141 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.851100 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt" event={"ID":"dfacf7de-fac7-47ee-84b7-4fe5a51e0b26","Type":"ContainerDied","Data":"8bb07615042f76ab3d3f65affda10a1a0f54826398a90f5b3415ccfff5fa44f5"} Nov 03 21:22:56.851272 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.851156 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt" Nov 03 21:22:56.871553 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.866784 5615 scope.go:122] "RemoveContainer" containerID="4c28ce2c3750835cd5e64e3f699c56a8f63b8812369a2a319683c28949b8200e" Nov 03 21:22:56.871553 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.867479 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/911bcfcb-bd59-4477-8b70-02f725d43ff7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/911bcfcb-bd59-4477-8b70-02f725d43ff7-plugins-config-volume\") pod \"911bcfcb-bd59-4477-8b70-02f725d43ff7\" (UID: \"911bcfcb-bd59-4477-8b70-02f725d43ff7\") " Nov 03 21:22:56.871553 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.867582 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/911bcfcb-bd59-4477-8b70-02f725d43ff7-kube-api-access-q5wr7\" (UniqueName: \"kubernetes.io/projected/911bcfcb-bd59-4477-8b70-02f725d43ff7-kube-api-access-q5wr7\") pod \"911bcfcb-bd59-4477-8b70-02f725d43ff7\" (UID: \"911bcfcb-bd59-4477-8b70-02f725d43ff7\") " Nov 03 21:22:56.871553 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:22:56.869980 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c28ce2c3750835cd5e64e3f699c56a8f63b8812369a2a319683c28949b8200e\": container with ID starting with 4c28ce2c3750835cd5e64e3f699c56a8f63b8812369a2a319683c28949b8200e not found: ID does not exist" containerID="4c28ce2c3750835cd5e64e3f699c56a8f63b8812369a2a319683c28949b8200e" Nov 03 21:22:56.871553 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.870085 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/911bcfcb-bd59-4477-8b70-02f725d43ff7-plugins-config-volume" pod "911bcfcb-bd59-4477-8b70-02f725d43ff7" (UID: "911bcfcb-bd59-4477-8b70-02f725d43ff7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:22:56.871553 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.870055 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c28ce2c3750835cd5e64e3f699c56a8f63b8812369a2a319683c28949b8200e"} err="failed to get container status \"4c28ce2c3750835cd5e64e3f699c56a8f63b8812369a2a319683c28949b8200e\": rpc error: code = NotFound desc = could not find container \"4c28ce2c3750835cd5e64e3f699c56a8f63b8812369a2a319683c28949b8200e\": container with ID starting with 4c28ce2c3750835cd5e64e3f699c56a8f63b8812369a2a319683c28949b8200e not found: ID does not exist" Nov 03 21:22:56.871553 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.870114 5615 scope.go:122] "RemoveContainer" containerID="da0724ab87045a3202c7b2267d7e0a78c187939893f8ed35ecceff2fa831f41a" Nov 03 21:22:56.889464 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.888173 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/911bcfcb-bd59-4477-8b70-02f725d43ff7-kube-api-access-q5wr7" pod "911bcfcb-bd59-4477-8b70-02f725d43ff7" (UID: "911bcfcb-bd59-4477-8b70-02f725d43ff7"). InnerVolumeSpecName "kube-api-access-q5wr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:22:56.898932 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.896611 5615 scope.go:122] "RemoveContainer" containerID="da0724ab87045a3202c7b2267d7e0a78c187939893f8ed35ecceff2fa831f41a" Nov 03 21:22:56.898932 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:22:56.898711 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da0724ab87045a3202c7b2267d7e0a78c187939893f8ed35ecceff2fa831f41a\": container with ID starting with da0724ab87045a3202c7b2267d7e0a78c187939893f8ed35ecceff2fa831f41a not found: ID does not exist" containerID="da0724ab87045a3202c7b2267d7e0a78c187939893f8ed35ecceff2fa831f41a" Nov 03 21:22:56.898932 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:56.898758 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da0724ab87045a3202c7b2267d7e0a78c187939893f8ed35ecceff2fa831f41a"} err="failed to get container status \"da0724ab87045a3202c7b2267d7e0a78c187939893f8ed35ecceff2fa831f41a\": rpc error: code = NotFound desc = could not find container \"da0724ab87045a3202c7b2267d7e0a78c187939893f8ed35ecceff2fa831f41a\": container with ID starting with da0724ab87045a3202c7b2267d7e0a78c187939893f8ed35ecceff2fa831f41a not found: ID does not exist" Nov 03 21:22:57.025034 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:57.002675 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/911bcfcb-bd59-4477-8b70-02f725d43ff7-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:57.025034 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:57.002712 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5wr7\" (UniqueName: \"kubernetes.io/projected/911bcfcb-bd59-4477-8b70-02f725d43ff7-kube-api-access-q5wr7\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:57.081205 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:57.057470 5615 patch_prober.go:29] interesting pod/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 21:22:57.081205 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:57.057533 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" podUID="d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 21:22:57.086484 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:57.085515 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt"] Nov 03 21:22:57.088709 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:57.088635 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-66513c-64b9d455c4-httpt"] Nov 03 21:22:57.188060 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:22:57.188025 5615 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911bcfcb_bd59_4477_8b70_02f725d43ff7.slice\": RecentStats: unable to find data in memory cache]" Nov 03 21:22:57.276571 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:57.276486 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-66513c-7765fcd666-6l4wb"] Nov 03 21:22:57.280665 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:57.280625 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-66513c-7765fcd666-6l4wb"] Nov 03 21:22:57.652395 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:57.652306 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="911bcfcb-bd59-4477-8b70-02f725d43ff7" path="/var/lib/kubelet/pods/911bcfcb-bd59-4477-8b70-02f725d43ff7/volumes" Nov 03 21:22:57.652789 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:57.652775 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfacf7de-fac7-47ee-84b7-4fe5a51e0b26" path="/var/lib/kubelet/pods/dfacf7de-fac7-47ee-84b7-4fe5a51e0b26/volumes" Nov 03 21:22:58.056044 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:58.056003 5615 patch_prober.go:29] interesting pod/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 21:22:58.056262 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:58.056059 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" podUID="d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 21:22:59.141884 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:59.141846 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:22:59.143009 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:59.142980 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:22:59.144917 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:22:59.144888 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:23:10.655485 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:23:10.655392 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:23:17.291556 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:17.291508 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr"] Nov 03 21:23:17.399573 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:17.399532 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh"] Nov 03 21:23:22.292348 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:22.292300 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr" podUID="a23823d8-ee6f-4b00-879c-0c05498338ae" containerName="payload-processing" containerID="cri-o://3f688a139a79ea764546782fcd995e944067aa33708eb8013f374dc1d4720ef0" gracePeriod=25 Nov 03 21:23:22.399995 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:22.399916 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh" podUID="ae01697d-2a38-41f0-8bcf-695b6ec9d967" containerName="payload-pre-processing" containerID="cri-o://81c60eb74f04a1b930c5b5d7cbbf4065db676651cfe27b773e8e8430982178ca" gracePeriod=25 Nov 03 21:23:22.546197 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:23:22.546166 5615 logging.go:55] [core] [Channel #3200 SubChannel #3202]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.114:50052", ServerName: "10.134.0.114:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.114:50052: connect: connection refused" Nov 03 21:23:22.579749 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:22.579717 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr" Nov 03 21:23:22.667032 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:22.666999 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a23823d8-ee6f-4b00-879c-0c05498338ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a23823d8-ee6f-4b00-879c-0c05498338ae-plugins-config-volume\") pod \"a23823d8-ee6f-4b00-879c-0c05498338ae\" (UID: \"a23823d8-ee6f-4b00-879c-0c05498338ae\") " Nov 03 21:23:22.667279 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:22.667115 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a23823d8-ee6f-4b00-879c-0c05498338ae-kube-api-access-fnjjb\" (UniqueName: \"kubernetes.io/projected/a23823d8-ee6f-4b00-879c-0c05498338ae-kube-api-access-fnjjb\") pod \"a23823d8-ee6f-4b00-879c-0c05498338ae\" (UID: \"a23823d8-ee6f-4b00-879c-0c05498338ae\") " Nov 03 21:23:22.667428 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:22.667403 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a23823d8-ee6f-4b00-879c-0c05498338ae-plugins-config-volume" pod "a23823d8-ee6f-4b00-879c-0c05498338ae" (UID: "a23823d8-ee6f-4b00-879c-0c05498338ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:23:22.667666 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:22.667650 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a23823d8-ee6f-4b00-879c-0c05498338ae-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:23:22.669481 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:22.669452 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23823d8-ee6f-4b00-879c-0c05498338ae-kube-api-access-fnjjb" pod "a23823d8-ee6f-4b00-879c-0c05498338ae" (UID: "a23823d8-ee6f-4b00-879c-0c05498338ae"). InnerVolumeSpecName "kube-api-access-fnjjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:23:22.691164 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:22.691137 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh" Nov 03 21:23:22.768560 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:22.768520 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae01697d-2a38-41f0-8bcf-695b6ec9d967-kube-api-access-s7259\" (UniqueName: \"kubernetes.io/projected/ae01697d-2a38-41f0-8bcf-695b6ec9d967-kube-api-access-s7259\") pod \"ae01697d-2a38-41f0-8bcf-695b6ec9d967\" (UID: \"ae01697d-2a38-41f0-8bcf-695b6ec9d967\") " Nov 03 21:23:22.768746 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:22.768608 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae01697d-2a38-41f0-8bcf-695b6ec9d967-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae01697d-2a38-41f0-8bcf-695b6ec9d967-plugins-config-volume\") pod \"ae01697d-2a38-41f0-8bcf-695b6ec9d967\" (UID: \"ae01697d-2a38-41f0-8bcf-695b6ec9d967\") " Nov 03 21:23:22.768999 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:22.768973 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae01697d-2a38-41f0-8bcf-695b6ec9d967-plugins-config-volume" pod "ae01697d-2a38-41f0-8bcf-695b6ec9d967" (UID: "ae01697d-2a38-41f0-8bcf-695b6ec9d967"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:23:22.769680 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:22.769639 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnjjb\" (UniqueName: \"kubernetes.io/projected/a23823d8-ee6f-4b00-879c-0c05498338ae-kube-api-access-fnjjb\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:23:22.769680 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:22.769669 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae01697d-2a38-41f0-8bcf-695b6ec9d967-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:23:22.770887 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:22.770864 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae01697d-2a38-41f0-8bcf-695b6ec9d967-kube-api-access-s7259" pod "ae01697d-2a38-41f0-8bcf-695b6ec9d967" (UID: "ae01697d-2a38-41f0-8bcf-695b6ec9d967"). InnerVolumeSpecName "kube-api-access-s7259". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:23:22.870497 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:22.870420 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7259\" (UniqueName: \"kubernetes.io/projected/ae01697d-2a38-41f0-8bcf-695b6ec9d967-kube-api-access-s7259\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:23:23.049139 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:23.049100 5615 generic.go:356] "Generic (PLEG): container finished" podID="a23823d8-ee6f-4b00-879c-0c05498338ae" containerID="3f688a139a79ea764546782fcd995e944067aa33708eb8013f374dc1d4720ef0" exitCode=0 Nov 03 21:23:23.049339 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:23.049164 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr" Nov 03 21:23:23.049339 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:23.049168 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr" event={"ID":"a23823d8-ee6f-4b00-879c-0c05498338ae","Type":"ContainerDied","Data":"3f688a139a79ea764546782fcd995e944067aa33708eb8013f374dc1d4720ef0"} Nov 03 21:23:23.049339 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:23.049224 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr" event={"ID":"a23823d8-ee6f-4b00-879c-0c05498338ae","Type":"ContainerDied","Data":"535d373db9c6c254b9f424cf353007da83e806ae6f319cc489b7a16c1328ce03"} Nov 03 21:23:23.049339 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:23.049249 5615 scope.go:122] "RemoveContainer" containerID="3f688a139a79ea764546782fcd995e944067aa33708eb8013f374dc1d4720ef0" Nov 03 21:23:23.050725 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:23.050700 5615 generic.go:356] "Generic (PLEG): container finished" podID="ae01697d-2a38-41f0-8bcf-695b6ec9d967" containerID="81c60eb74f04a1b930c5b5d7cbbf4065db676651cfe27b773e8e8430982178ca" exitCode=0 Nov 03 21:23:23.050822 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:23.050730 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh" event={"ID":"ae01697d-2a38-41f0-8bcf-695b6ec9d967","Type":"ContainerDied","Data":"81c60eb74f04a1b930c5b5d7cbbf4065db676651cfe27b773e8e8430982178ca"} Nov 03 21:23:23.050822 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:23.050758 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh" Nov 03 21:23:23.050822 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:23.050781 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh" event={"ID":"ae01697d-2a38-41f0-8bcf-695b6ec9d967","Type":"ContainerDied","Data":"0d3ca6acfec4421b92273505a316435c1504bd24a79fd10b3c322c84b401eb74"} Nov 03 21:23:23.059235 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:23.059212 5615 scope.go:122] "RemoveContainer" containerID="3f688a139a79ea764546782fcd995e944067aa33708eb8013f374dc1d4720ef0" Nov 03 21:23:23.059527 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:23:23.059505 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f688a139a79ea764546782fcd995e944067aa33708eb8013f374dc1d4720ef0\": container with ID starting with 3f688a139a79ea764546782fcd995e944067aa33708eb8013f374dc1d4720ef0 not found: ID does not exist" containerID="3f688a139a79ea764546782fcd995e944067aa33708eb8013f374dc1d4720ef0" Nov 03 21:23:23.059643 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:23.059535 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f688a139a79ea764546782fcd995e944067aa33708eb8013f374dc1d4720ef0"} err="failed to get container status \"3f688a139a79ea764546782fcd995e944067aa33708eb8013f374dc1d4720ef0\": rpc error: code = NotFound desc = could not find container \"3f688a139a79ea764546782fcd995e944067aa33708eb8013f374dc1d4720ef0\": container with ID starting with 3f688a139a79ea764546782fcd995e944067aa33708eb8013f374dc1d4720ef0 not found: ID does not exist" Nov 03 21:23:23.059643 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:23.059557 5615 scope.go:122] "RemoveContainer" containerID="81c60eb74f04a1b930c5b5d7cbbf4065db676651cfe27b773e8e8430982178ca" Nov 03 21:23:23.070954 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:23.070930 5615 scope.go:122] "RemoveContainer" containerID="81c60eb74f04a1b930c5b5d7cbbf4065db676651cfe27b773e8e8430982178ca" Nov 03 21:23:23.071344 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:23:23.071316 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81c60eb74f04a1b930c5b5d7cbbf4065db676651cfe27b773e8e8430982178ca\": container with ID starting with 81c60eb74f04a1b930c5b5d7cbbf4065db676651cfe27b773e8e8430982178ca not found: ID does not exist" containerID="81c60eb74f04a1b930c5b5d7cbbf4065db676651cfe27b773e8e8430982178ca" Nov 03 21:23:23.071432 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:23.071351 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c60eb74f04a1b930c5b5d7cbbf4065db676651cfe27b773e8e8430982178ca"} err="failed to get container status \"81c60eb74f04a1b930c5b5d7cbbf4065db676651cfe27b773e8e8430982178ca\": rpc error: code = NotFound desc = could not find container \"81c60eb74f04a1b930c5b5d7cbbf4065db676651cfe27b773e8e8430982178ca\": container with ID starting with 81c60eb74f04a1b930c5b5d7cbbf4065db676651cfe27b773e8e8430982178ca not found: ID does not exist" Nov 03 21:23:23.206327 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:23.206243 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr"] Nov 03 21:23:23.209682 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:23.209646 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr"] Nov 03 21:23:23.314044 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:23.314004 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh"] Nov 03 21:23:23.316766 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:23.316735 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh"] Nov 03 21:23:23.546492 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:23.546440 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-3e1b49-55c79f4c4d-vr8lr" podUID="a23823d8-ee6f-4b00-879c-0c05498338ae" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.115:50052\" within 1s: context deadline exceeded" Nov 03 21:23:23.546659 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:23:23.546541 5615 logging.go:55] [core] [Channel #3199 SubChannel #3201]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.115:50052", ServerName: "10.134.0.115:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.115:50052: operation was canceled" Nov 03 21:23:23.546659 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:23.546451 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3e1b49-cfb488b75-c6xwh" podUID="ae01697d-2a38-41f0-8bcf-695b6ec9d967" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.114:50052\" within 1s: context deadline exceeded" Nov 03 21:23:23.652615 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:23.652570 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a23823d8-ee6f-4b00-879c-0c05498338ae" path="/var/lib/kubelet/pods/a23823d8-ee6f-4b00-879c-0c05498338ae/volumes" Nov 03 21:23:23.653193 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:23.653168 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae01697d-2a38-41f0-8bcf-695b6ec9d967" path="/var/lib/kubelet/pods/ae01697d-2a38-41f0-8bcf-695b6ec9d967/volumes" Nov 03 21:23:26.642953 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:26.642917 5615 scope.go:122] "RemoveContainer" containerID="2a36ca7d8124bb12bda5b8dbe505174f4caef277afc7f5b17773112434b8bfbb" Nov 03 21:23:34.388974 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:34.388933 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498"] Nov 03 21:23:34.389482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:34.389160 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498" podUID="2d28992b-9e00-4787-aa1c-3af045d450ea" containerName="maas-api" containerID="cri-o://933f32ef523622bc0e044c94eb8a204c72c8626a56ac1781b6f0663f9aa60eac" gracePeriod=30 Nov 03 21:23:34.698540 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:34.698513 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498" Nov 03 21:23:34.839631 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:34.839582 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d28992b-9e00-4787-aa1c-3af045d450ea-kube-api-access-hkdwj\" (UniqueName: \"kubernetes.io/projected/2d28992b-9e00-4787-aa1c-3af045d450ea-kube-api-access-hkdwj\") pod \"2d28992b-9e00-4787-aa1c-3af045d450ea\" (UID: \"2d28992b-9e00-4787-aa1c-3af045d450ea\") " Nov 03 21:23:34.839831 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:34.839662 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d28992b-9e00-4787-aa1c-3af045d450ea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d28992b-9e00-4787-aa1c-3af045d450ea-openshift-service-ca\") pod \"2d28992b-9e00-4787-aa1c-3af045d450ea\" (UID: \"2d28992b-9e00-4787-aa1c-3af045d450ea\") " Nov 03 21:23:34.839831 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:34.839775 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d28992b-9e00-4787-aa1c-3af045d450ea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d28992b-9e00-4787-aa1c-3af045d450ea-maas-api-tls\") pod \"2d28992b-9e00-4787-aa1c-3af045d450ea\" (UID: \"2d28992b-9e00-4787-aa1c-3af045d450ea\") " Nov 03 21:23:34.839831 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:34.839809 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d28992b-9e00-4787-aa1c-3af045d450ea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d28992b-9e00-4787-aa1c-3af045d450ea-metrics-certs\") pod \"2d28992b-9e00-4787-aa1c-3af045d450ea\" (UID: \"2d28992b-9e00-4787-aa1c-3af045d450ea\") " Nov 03 21:23:34.840024 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:34.839995 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d28992b-9e00-4787-aa1c-3af045d450ea-openshift-service-ca" pod "2d28992b-9e00-4787-aa1c-3af045d450ea" (UID: "2d28992b-9e00-4787-aa1c-3af045d450ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:23:34.840463 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:34.840429 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d28992b-9e00-4787-aa1c-3af045d450ea-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:23:34.842059 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:34.842035 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d28992b-9e00-4787-aa1c-3af045d450ea-metrics-certs" pod "2d28992b-9e00-4787-aa1c-3af045d450ea" (UID: "2d28992b-9e00-4787-aa1c-3af045d450ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:23:34.842320 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:34.842298 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d28992b-9e00-4787-aa1c-3af045d450ea-kube-api-access-hkdwj" pod "2d28992b-9e00-4787-aa1c-3af045d450ea" (UID: "2d28992b-9e00-4787-aa1c-3af045d450ea"). InnerVolumeSpecName "kube-api-access-hkdwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:23:34.845222 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:34.845196 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d28992b-9e00-4787-aa1c-3af045d450ea-maas-api-tls" pod "2d28992b-9e00-4787-aa1c-3af045d450ea" (UID: "2d28992b-9e00-4787-aa1c-3af045d450ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:23:34.858803 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:34.858756 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-f6d7b55fb-k5rwz"] Nov 03 21:23:34.859004 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:34.858961 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-f6d7b55fb-k5rwz" podUID="ee45aa1b-3c2e-4721-bf25-28631bcb0110" containerName="payload-pre-processing" containerID="cri-o://350b82da0033d2a2a7e9876da1ee7ddc46057edf41807772bd0a63b809193b07" gracePeriod=30 Nov 03 21:23:34.941570 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:34.941484 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d28992b-9e00-4787-aa1c-3af045d450ea-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:23:34.941570 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:34.941516 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d28992b-9e00-4787-aa1c-3af045d450ea-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:23:34.941570 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:34.941538 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkdwj\" (UniqueName: \"kubernetes.io/projected/2d28992b-9e00-4787-aa1c-3af045d450ea-kube-api-access-hkdwj\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:23:35.115531 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.115502 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-f6d7b55fb-k5rwz" Nov 03 21:23:35.245744 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.245702 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee45aa1b-3c2e-4721-bf25-28631bcb0110-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee45aa1b-3c2e-4721-bf25-28631bcb0110-plugins-config-volume\") pod \"ee45aa1b-3c2e-4721-bf25-28631bcb0110\" (UID: \"ee45aa1b-3c2e-4721-bf25-28631bcb0110\") " Nov 03 21:23:35.245948 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.245901 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee45aa1b-3c2e-4721-bf25-28631bcb0110-kube-api-access-cf8hq\" (UniqueName: \"kubernetes.io/projected/ee45aa1b-3c2e-4721-bf25-28631bcb0110-kube-api-access-cf8hq\") pod \"ee45aa1b-3c2e-4721-bf25-28631bcb0110\" (UID: \"ee45aa1b-3c2e-4721-bf25-28631bcb0110\") " Nov 03 21:23:35.246139 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.246105 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee45aa1b-3c2e-4721-bf25-28631bcb0110-plugins-config-volume" pod "ee45aa1b-3c2e-4721-bf25-28631bcb0110" (UID: "ee45aa1b-3c2e-4721-bf25-28631bcb0110"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:23:35.246430 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.246410 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee45aa1b-3c2e-4721-bf25-28631bcb0110-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:23:35.247942 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.247921 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee45aa1b-3c2e-4721-bf25-28631bcb0110-kube-api-access-cf8hq" pod "ee45aa1b-3c2e-4721-bf25-28631bcb0110" (UID: "ee45aa1b-3c2e-4721-bf25-28631bcb0110"). InnerVolumeSpecName "kube-api-access-cf8hq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:23:35.276380 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.276335 5615 generic.go:356] "Generic (PLEG): container finished" podID="2d28992b-9e00-4787-aa1c-3af045d450ea" containerID="933f32ef523622bc0e044c94eb8a204c72c8626a56ac1781b6f0663f9aa60eac" exitCode=0 Nov 03 21:23:35.276579 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.276425 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498" Nov 03 21:23:35.276579 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.276467 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498" event={"ID":"2d28992b-9e00-4787-aa1c-3af045d450ea","Type":"ContainerDied","Data":"933f32ef523622bc0e044c94eb8a204c72c8626a56ac1781b6f0663f9aa60eac"} Nov 03 21:23:35.276579 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.276541 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498" event={"ID":"2d28992b-9e00-4787-aa1c-3af045d450ea","Type":"ContainerDied","Data":"3202a3180d312528cc2a1418524f3475f7f0664f0782f864d0510a69e9b186cb"} Nov 03 21:23:35.276750 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.276570 5615 scope.go:122] "RemoveContainer" containerID="933f32ef523622bc0e044c94eb8a204c72c8626a56ac1781b6f0663f9aa60eac" Nov 03 21:23:35.283587 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.283273 5615 generic.go:356] "Generic (PLEG): container finished" podID="ee45aa1b-3c2e-4721-bf25-28631bcb0110" containerID="350b82da0033d2a2a7e9876da1ee7ddc46057edf41807772bd0a63b809193b07" exitCode=0 Nov 03 21:23:35.283587 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.283357 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-f6d7b55fb-k5rwz" Nov 03 21:23:35.283587 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.283373 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-f6d7b55fb-k5rwz" event={"ID":"ee45aa1b-3c2e-4721-bf25-28631bcb0110","Type":"ContainerDied","Data":"350b82da0033d2a2a7e9876da1ee7ddc46057edf41807772bd0a63b809193b07"} Nov 03 21:23:35.283587 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.283413 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-f6d7b55fb-k5rwz" event={"ID":"ee45aa1b-3c2e-4721-bf25-28631bcb0110","Type":"ContainerDied","Data":"fd49299d2723506d1898a0b8b8c58588132d4f17c796a8d81ee4b7a32b75fd2b"} Nov 03 21:23:35.294667 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.294646 5615 scope.go:122] "RemoveContainer" containerID="933f32ef523622bc0e044c94eb8a204c72c8626a56ac1781b6f0663f9aa60eac" Nov 03 21:23:35.294995 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:23:35.294968 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"933f32ef523622bc0e044c94eb8a204c72c8626a56ac1781b6f0663f9aa60eac\": container with ID starting with 933f32ef523622bc0e044c94eb8a204c72c8626a56ac1781b6f0663f9aa60eac not found: ID does not exist" containerID="933f32ef523622bc0e044c94eb8a204c72c8626a56ac1781b6f0663f9aa60eac" Nov 03 21:23:35.295155 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.295000 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"933f32ef523622bc0e044c94eb8a204c72c8626a56ac1781b6f0663f9aa60eac"} err="failed to get container status \"933f32ef523622bc0e044c94eb8a204c72c8626a56ac1781b6f0663f9aa60eac\": rpc error: code = NotFound desc = could not find container \"933f32ef523622bc0e044c94eb8a204c72c8626a56ac1781b6f0663f9aa60eac\": container with ID starting with 933f32ef523622bc0e044c94eb8a204c72c8626a56ac1781b6f0663f9aa60eac not found: ID does not exist" Nov 03 21:23:35.295155 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.295019 5615 scope.go:122] "RemoveContainer" containerID="350b82da0033d2a2a7e9876da1ee7ddc46057edf41807772bd0a63b809193b07" Nov 03 21:23:35.306936 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.306903 5615 scope.go:122] "RemoveContainer" containerID="350b82da0033d2a2a7e9876da1ee7ddc46057edf41807772bd0a63b809193b07" Nov 03 21:23:35.307426 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:23:35.307364 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"350b82da0033d2a2a7e9876da1ee7ddc46057edf41807772bd0a63b809193b07\": container with ID starting with 350b82da0033d2a2a7e9876da1ee7ddc46057edf41807772bd0a63b809193b07 not found: ID does not exist" containerID="350b82da0033d2a2a7e9876da1ee7ddc46057edf41807772bd0a63b809193b07" Nov 03 21:23:35.307426 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.307403 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"350b82da0033d2a2a7e9876da1ee7ddc46057edf41807772bd0a63b809193b07"} err="failed to get container status \"350b82da0033d2a2a7e9876da1ee7ddc46057edf41807772bd0a63b809193b07\": rpc error: code = NotFound desc = could not find container \"350b82da0033d2a2a7e9876da1ee7ddc46057edf41807772bd0a63b809193b07\": container with ID starting with 350b82da0033d2a2a7e9876da1ee7ddc46057edf41807772bd0a63b809193b07 not found: ID does not exist" Nov 03 21:23:35.376033 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.371523 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf8hq\" (UniqueName: \"kubernetes.io/projected/ee45aa1b-3c2e-4721-bf25-28631bcb0110-kube-api-access-cf8hq\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:23:35.458292 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.458247 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-f6d7b55fb-k5rwz"] Nov 03 21:23:35.464630 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.464585 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-f6d7b55fb-k5rwz"] Nov 03 21:23:35.570037 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.569944 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498"] Nov 03 21:23:35.574140 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.574104 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-812ef9-77c58ff8b6-sm498"] Nov 03 21:23:35.652399 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.652367 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d28992b-9e00-4787-aa1c-3af045d450ea" path="/var/lib/kubelet/pods/2d28992b-9e00-4787-aa1c-3af045d450ea/volumes" Nov 03 21:23:35.652818 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.652805 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee45aa1b-3c2e-4721-bf25-28631bcb0110" path="/var/lib/kubelet/pods/ee45aa1b-3c2e-4721-bf25-28631bcb0110/volumes" Nov 03 21:23:35.934375 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.934290 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h"] Nov 03 21:23:35.934936 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.934919 5615 cpu_manager.go:414] "containerMap: removing container" podUID="ae01697d-2a38-41f0-8bcf-695b6ec9d967" containerName="payload-pre-processing" Nov 03 21:23:35.934988 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.934938 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae01697d-2a38-41f0-8bcf-695b6ec9d967" containerName="payload-pre-processing" Nov 03 21:23:35.934988 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.934963 5615 cpu_manager.go:414] "containerMap: removing container" podUID="dfacf7de-fac7-47ee-84b7-4fe5a51e0b26" containerName="maas-api" Nov 03 21:23:35.934988 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.934969 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfacf7de-fac7-47ee-84b7-4fe5a51e0b26" containerName="maas-api" Nov 03 21:23:35.934988 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.934979 5615 cpu_manager.go:414] "containerMap: removing container" podUID="ee45aa1b-3c2e-4721-bf25-28631bcb0110" containerName="payload-pre-processing" Nov 03 21:23:35.934988 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.934984 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee45aa1b-3c2e-4721-bf25-28631bcb0110" containerName="payload-pre-processing" Nov 03 21:23:35.935204 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.935003 5615 cpu_manager.go:414] "containerMap: removing container" podUID="2d28992b-9e00-4787-aa1c-3af045d450ea" containerName="maas-api" Nov 03 21:23:35.935204 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.935008 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d28992b-9e00-4787-aa1c-3af045d450ea" containerName="maas-api" Nov 03 21:23:35.935204 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.935015 5615 cpu_manager.go:414] "containerMap: removing container" podUID="911bcfcb-bd59-4477-8b70-02f725d43ff7" containerName="payload-processing" Nov 03 21:23:35.935204 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.935020 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="911bcfcb-bd59-4477-8b70-02f725d43ff7" containerName="payload-processing" Nov 03 21:23:35.935204 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.935041 5615 cpu_manager.go:414] "containerMap: removing container" podUID="a23823d8-ee6f-4b00-879c-0c05498338ae" containerName="payload-processing" Nov 03 21:23:35.935204 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.935046 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a23823d8-ee6f-4b00-879c-0c05498338ae" containerName="payload-processing" Nov 03 21:23:35.935204 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.935143 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="a23823d8-ee6f-4b00-879c-0c05498338ae" containerName="payload-processing" Nov 03 21:23:35.935204 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.935155 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae01697d-2a38-41f0-8bcf-695b6ec9d967" containerName="payload-pre-processing" Nov 03 21:23:35.935204 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.935164 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfacf7de-fac7-47ee-84b7-4fe5a51e0b26" containerName="maas-api" Nov 03 21:23:35.935204 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.935171 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee45aa1b-3c2e-4721-bf25-28631bcb0110" containerName="payload-pre-processing" Nov 03 21:23:35.935204 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.935180 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d28992b-9e00-4787-aa1c-3af045d450ea" containerName="maas-api" Nov 03 21:23:35.935204 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.935185 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="911bcfcb-bd59-4477-8b70-02f725d43ff7" containerName="payload-processing" Nov 03 21:23:35.940020 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.939996 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h" Nov 03 21:23:35.986101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.986033 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h\" (UID: \"9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h" Nov 03 21:23:35.986308 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.986152 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h\" (UID: \"9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h" Nov 03 21:23:35.986308 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.986206 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km9np\" (UniqueName: \"kubernetes.io/projected/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-kube-api-access-km9np\") pod \"maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h\" (UID: \"9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h" Nov 03 21:23:35.986435 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.986355 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-metrics-certs\") pod \"maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h\" (UID: \"9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h" Nov 03 21:23:35.993583 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:35.993555 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h"] Nov 03 21:23:36.042844 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:36.042813 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-3d317d-serving-cert\"" Nov 03 21:23:36.087497 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:36.087445 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km9np\" (UniqueName: \"kubernetes.io/projected/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-kube-api-access-km9np\") pod \"maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h\" (UID: \"9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h" Nov 03 21:23:36.087690 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:36.087590 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-metrics-certs\") pod \"maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h\" (UID: \"9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h" Nov 03 21:23:36.087819 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:36.087800 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h\" (UID: \"9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h" Nov 03 21:23:36.087895 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:36.087834 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h\" (UID: \"9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h" Nov 03 21:23:36.087954 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:23:36.087889 5615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-serving-cert: secret "maas-api-e2e-praxis-a-3d317d-serving-cert" not found Nov 03 21:23:36.088010 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:23:36.087969 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-maas-api-tls podName:9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a nodeName:}" failed. No retries permitted until 2026-11-03 21:23:36.587948591 +0000 UTC m=+2111.055727323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-maas-api-tls") pod "maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h" (UID: "9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a") : secret "maas-api-e2e-praxis-a-3d317d-serving-cert" not found Nov 03 21:23:36.088668 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:36.088641 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h\" (UID: \"9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h" Nov 03 21:23:36.090672 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:36.090648 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-metrics-certs\") pod \"maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h\" (UID: \"9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h" Nov 03 21:23:36.148712 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:36.148671 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km9np\" (UniqueName: \"kubernetes.io/projected/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-kube-api-access-km9np\") pod \"maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h\" (UID: \"9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h" Nov 03 21:23:36.616824 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:36.616782 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h\" (UID: \"9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h" Nov 03 21:23:36.619430 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:36.619407 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h\" (UID: \"9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h" Nov 03 21:23:36.854639 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:36.854594 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h" Nov 03 21:23:37.227161 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:37.227133 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h"] Nov 03 21:23:37.228785 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:23:37.228748 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5d0fab_d0db_4bcd_94e6_8bfea3d2d14a.slice/crio-5693ca1728162b10f23e66aa103010093ade37f44516dbe1633b2993ef537528 WatchSource:0}: Error finding container 5693ca1728162b10f23e66aa103010093ade37f44516dbe1633b2993ef537528: Status 404 returned error can't find the container with id 5693ca1728162b10f23e66aa103010093ade37f44516dbe1633b2993ef537528 Nov 03 21:23:37.302787 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:37.302751 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h" event={"ID":"9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a","Type":"ContainerStarted","Data":"5693ca1728162b10f23e66aa103010093ade37f44516dbe1633b2993ef537528"} Nov 03 21:23:38.311024 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:38.310980 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h" event={"ID":"9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a","Type":"ContainerStarted","Data":"a0121807123bb57235525e54d0458f11ec21e91c7a88ad75be84a887b929a5a6"} Nov 03 21:23:38.311443 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:38.311049 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h" Nov 03 21:23:38.442613 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:38.442512 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h" podStartSLOduration=2.5201698329999997 podStartE2EDuration="3.442496805s" totalImagesPullingTime="922.326972ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:23:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:23:38.441478634 +0000 UTC m=+2112.909257362" watchObservedRunningTime="2026-11-03 21:23:38.442496805 +0000 UTC m=+2112.910275527" Nov 03 21:23:40.619924 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:23:40.619885 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:23:44.325890 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:44.325854 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h" Nov 03 21:23:48.461660 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.461618 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x"] Nov 03 21:23:48.465789 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.465766 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:48.517476 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.517442 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-b5c926-openshift-default-dockercfg-6lgbr\"" Nov 03 21:23:48.605323 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.605277 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-workload-certs\") pod \"e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:48.605631 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.605605 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09573228-1c0e-478a-b836-7b46b30e6c99-istio-token\") pod \"e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:48.605808 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.605791 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-istio-envoy\") pod \"e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:48.605944 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.605929 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-istio-data\") pod \"e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:48.606097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.606063 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-credential-socket\") pod \"e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:48.606215 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.606200 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09573228-1c0e-478a-b836-7b46b30e6c99-istio-podinfo\") pod \"e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:48.606361 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.606344 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-workload-socket\") pod \"e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:48.606516 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.606499 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2pb5\" (UniqueName: \"kubernetes.io/projected/09573228-1c0e-478a-b836-7b46b30e6c99-kube-api-access-z2pb5\") pod \"e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:48.606726 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.606708 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09573228-1c0e-478a-b836-7b46b30e6c99-istiod-ca-cert\") pod \"e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:48.695288 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.695244 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x"] Nov 03 21:23:48.711710 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.711590 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2pb5\" (UniqueName: \"kubernetes.io/projected/09573228-1c0e-478a-b836-7b46b30e6c99-kube-api-access-z2pb5\") pod \"e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:48.713602 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.713565 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09573228-1c0e-478a-b836-7b46b30e6c99-istiod-ca-cert\") pod \"e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:48.713954 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.713900 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-workload-certs\") pod \"e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:48.714189 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.714170 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09573228-1c0e-478a-b836-7b46b30e6c99-istio-token\") pod \"e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:48.714474 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.714446 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-istio-envoy\") pod \"e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:48.714695 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.714630 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-istio-data\") pod \"e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:48.714905 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.714885 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-credential-socket\") pod \"e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:48.715050 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.715032 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09573228-1c0e-478a-b836-7b46b30e6c99-istio-podinfo\") pod \"e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:48.715326 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.715308 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-workload-socket\") pod \"e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:48.715966 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.715907 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-workload-socket\") pod \"e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:48.716409 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.716382 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-credential-socket\") pod \"e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:48.718158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.716679 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-istio-data\") pod \"e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:48.718729 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.718683 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09573228-1c0e-478a-b836-7b46b30e6c99-istiod-ca-cert\") pod \"e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:48.718867 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.718842 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-workload-certs\") pod \"e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:48.731121 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.730995 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-istio-envoy\") pod \"e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:48.733262 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.733224 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09573228-1c0e-478a-b836-7b46b30e6c99-istio-podinfo\") pod \"e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:48.770449 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.770414 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2pb5\" (UniqueName: \"kubernetes.io/projected/09573228-1c0e-478a-b836-7b46b30e6c99-kube-api-access-z2pb5\") pod \"e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:48.773336 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.773311 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09573228-1c0e-478a-b836-7b46b30e6c99-istio-token\") pod \"e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:48.793112 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:48.781760 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:49.236170 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:49.236140 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x"] Nov 03 21:23:49.238039 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:23:49.238010 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09573228_1c0e_478a_b836_7b46b30e6c99.slice/crio-3eebcc7fc1944e6a3ddd9e0a0cfa7b52203445574a47a114f19e2a61bb8a8043 WatchSource:0}: Error finding container 3eebcc7fc1944e6a3ddd9e0a0cfa7b52203445574a47a114f19e2a61bb8a8043: Status 404 returned error can't find the container with id 3eebcc7fc1944e6a3ddd9e0a0cfa7b52203445574a47a114f19e2a61bb8a8043 Nov 03 21:23:49.240763 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:49.240731 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:23:49.240829 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:49.240799 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:23:49.240866 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:49.240829 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:23:49.434489 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:49.434449 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" event={"ID":"09573228-1c0e-478a-b836-7b46b30e6c99","Type":"ContainerStarted","Data":"6a77bdf86d0c69ccda26c60cf9cc5dc9b8646ac2c0c2f83ef912dc04ee6cfef2"} Nov 03 21:23:49.434672 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:49.434506 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" event={"ID":"09573228-1c0e-478a-b836-7b46b30e6c99","Type":"ContainerStarted","Data":"3eebcc7fc1944e6a3ddd9e0a0cfa7b52203445574a47a114f19e2a61bb8a8043"} Nov 03 21:23:49.533224 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:49.533140 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n"] Nov 03 21:23:49.533647 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:49.533339 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n" podUID="d22b9636-a144-4ba6-b538-d22b253c8908" containerName="maas-api" containerID="cri-o://78dddc2662eec18339dc157b8b562fab9323ffb48b31f2481c54abf4499bc173" gracePeriod=30 Nov 03 21:23:49.550222 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:49.550172 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" podStartSLOduration=1.550157204 podStartE2EDuration="1.550157204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:23:49.549116869 +0000 UTC m=+2124.016895592" watchObservedRunningTime="2026-11-03 21:23:49.550157204 +0000 UTC m=+2124.017935926" Nov 03 21:23:49.844907 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:49.844689 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:49.880850 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:49.880822 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n" Nov 03 21:23:50.048388 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:50.048341 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d22b9636-a144-4ba6-b538-d22b253c8908-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d22b9636-a144-4ba6-b538-d22b253c8908-openshift-service-ca\") pod \"d22b9636-a144-4ba6-b538-d22b253c8908\" (UID: \"d22b9636-a144-4ba6-b538-d22b253c8908\") " Nov 03 21:23:50.048576 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:50.048510 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d22b9636-a144-4ba6-b538-d22b253c8908-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d22b9636-a144-4ba6-b538-d22b253c8908-metrics-certs\") pod \"d22b9636-a144-4ba6-b538-d22b253c8908\" (UID: \"d22b9636-a144-4ba6-b538-d22b253c8908\") " Nov 03 21:23:50.048576 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:50.048546 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d22b9636-a144-4ba6-b538-d22b253c8908-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d22b9636-a144-4ba6-b538-d22b253c8908-maas-api-tls\") pod \"d22b9636-a144-4ba6-b538-d22b253c8908\" (UID: \"d22b9636-a144-4ba6-b538-d22b253c8908\") " Nov 03 21:23:50.048703 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:50.048692 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d22b9636-a144-4ba6-b538-d22b253c8908-kube-api-access-62q2t\" (UniqueName: \"kubernetes.io/projected/d22b9636-a144-4ba6-b538-d22b253c8908-kube-api-access-62q2t\") pod \"d22b9636-a144-4ba6-b538-d22b253c8908\" (UID: \"d22b9636-a144-4ba6-b538-d22b253c8908\") " Nov 03 21:23:50.048762 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:50.048738 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d22b9636-a144-4ba6-b538-d22b253c8908-openshift-service-ca" pod "d22b9636-a144-4ba6-b538-d22b253c8908" (UID: "d22b9636-a144-4ba6-b538-d22b253c8908"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:23:50.049245 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:50.049218 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d22b9636-a144-4ba6-b538-d22b253c8908-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:23:50.051095 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:50.051042 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d22b9636-a144-4ba6-b538-d22b253c8908-kube-api-access-62q2t" pod "d22b9636-a144-4ba6-b538-d22b253c8908" (UID: "d22b9636-a144-4ba6-b538-d22b253c8908"). InnerVolumeSpecName "kube-api-access-62q2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:23:50.051417 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:50.051397 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d22b9636-a144-4ba6-b538-d22b253c8908-metrics-certs" pod "d22b9636-a144-4ba6-b538-d22b253c8908" (UID: "d22b9636-a144-4ba6-b538-d22b253c8908"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:23:50.051619 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:50.051595 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d22b9636-a144-4ba6-b538-d22b253c8908-maas-api-tls" pod "d22b9636-a144-4ba6-b538-d22b253c8908" (UID: "d22b9636-a144-4ba6-b538-d22b253c8908"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:23:50.156408 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:50.156290 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d22b9636-a144-4ba6-b538-d22b253c8908-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:23:50.156408 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:50.156336 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d22b9636-a144-4ba6-b538-d22b253c8908-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:23:50.156408 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:50.156350 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62q2t\" (UniqueName: \"kubernetes.io/projected/d22b9636-a144-4ba6-b538-d22b253c8908-kube-api-access-62q2t\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:23:50.476151 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:50.475311 5615 generic.go:356] "Generic (PLEG): container finished" podID="d22b9636-a144-4ba6-b538-d22b253c8908" containerID="78dddc2662eec18339dc157b8b562fab9323ffb48b31f2481c54abf4499bc173" exitCode=0 Nov 03 21:23:50.479667 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:50.476694 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n" Nov 03 21:23:50.492101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:50.489156 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n" event={"ID":"d22b9636-a144-4ba6-b538-d22b253c8908","Type":"ContainerDied","Data":"78dddc2662eec18339dc157b8b562fab9323ffb48b31f2481c54abf4499bc173"} Nov 03 21:23:50.492101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:50.489252 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n" event={"ID":"d22b9636-a144-4ba6-b538-d22b253c8908","Type":"ContainerDied","Data":"e6a187097e5c39a3336eeb9d67ed045589c8c9505f1abb47d0f8d3157a9b9813"} Nov 03 21:23:50.492101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:50.489291 5615 scope.go:122] "RemoveContainer" containerID="78dddc2662eec18339dc157b8b562fab9323ffb48b31f2481c54abf4499bc173" Nov 03 21:23:50.558017 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:50.557850 5615 scope.go:122] "RemoveContainer" containerID="78dddc2662eec18339dc157b8b562fab9323ffb48b31f2481c54abf4499bc173" Nov 03 21:23:50.558440 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:23:50.558377 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78dddc2662eec18339dc157b8b562fab9323ffb48b31f2481c54abf4499bc173\": container with ID starting with 78dddc2662eec18339dc157b8b562fab9323ffb48b31f2481c54abf4499bc173 not found: ID does not exist" containerID="78dddc2662eec18339dc157b8b562fab9323ffb48b31f2481c54abf4499bc173" Nov 03 21:23:50.558440 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:50.558409 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78dddc2662eec18339dc157b8b562fab9323ffb48b31f2481c54abf4499bc173"} err="failed to get container status \"78dddc2662eec18339dc157b8b562fab9323ffb48b31f2481c54abf4499bc173\": rpc error: code = NotFound desc = could not find container \"78dddc2662eec18339dc157b8b562fab9323ffb48b31f2481c54abf4499bc173\": container with ID starting with 78dddc2662eec18339dc157b8b562fab9323ffb48b31f2481c54abf4499bc173 not found: ID does not exist" Nov 03 21:23:50.651991 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:50.651953 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n"] Nov 03 21:23:50.699749 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:50.699707 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3e1b49-56bd67d7d4-5bw7n"] Nov 03 21:23:50.794698 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:50.794668 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:51.500168 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:51.500135 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:51.501788 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:51.501764 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:23:51.664436 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:23:51.664401 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d22b9636-a144-4ba6-b538-d22b253c8908" path="/var/lib/kubelet/pods/d22b9636-a144-4ba6-b538-d22b253c8908/volumes" Nov 03 21:24:09.391576 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:09.391535 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk"] Nov 03 21:24:09.392066 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:09.391794 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" podUID="c34e4dc8-c666-4cf5-9a18-7fafb9607383" containerName="istio-proxy" containerID="cri-o://d85831cf86d0d2f19fbb138b983e13c980a884854f41b19c840e2503d10c6de2" gracePeriod=30 Nov 03 21:24:09.708004 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:09.707784 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh"] Nov 03 21:24:09.708237 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:09.708028 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" podUID="f5382faa-4094-4acf-a7e8-4538976acfb8" containerName="istio-proxy" containerID="cri-o://7a7af3bae7574fe7c19d468a021dc7bed857c50238383024db72ba5990a8ba80" gracePeriod=30 Nov 03 21:24:10.620262 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:24:10.620218 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:24:11.967315 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:11.967276 5615 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:24:11.967712 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:11.967339 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" podUID="f5382faa-4094-4acf-a7e8-4538976acfb8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:24:14.663961 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.663889 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:24:14.735688 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.735654 5615 generic.go:356] "Generic (PLEG): container finished" podID="c34e4dc8-c666-4cf5-9a18-7fafb9607383" containerID="d85831cf86d0d2f19fbb138b983e13c980a884854f41b19c840e2503d10c6de2" exitCode=0 Nov 03 21:24:14.735823 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.735702 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" Nov 03 21:24:14.735823 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.735731 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" event={"ID":"c34e4dc8-c666-4cf5-9a18-7fafb9607383","Type":"ContainerDied","Data":"d85831cf86d0d2f19fbb138b983e13c980a884854f41b19c840e2503d10c6de2"} Nov 03 21:24:14.735823 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.735787 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk" event={"ID":"c34e4dc8-c666-4cf5-9a18-7fafb9607383","Type":"ContainerDied","Data":"b83f6f020cb75a0dcbadfcb0279e5677d341f8e04ace39428fabb276f30e5199"} Nov 03 21:24:14.735823 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.735812 5615 scope.go:122] "RemoveContainer" containerID="d85831cf86d0d2f19fbb138b983e13c980a884854f41b19c840e2503d10c6de2" Nov 03 21:24:14.784291 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.784255 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istio-token\" (UniqueName: \"kubernetes.io/projected/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istio-token\") pod \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " Nov 03 21:24:14.784472 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.784337 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-credential-socket\") pod \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " Nov 03 21:24:14.784472 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.784371 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istio-data\") pod \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " Nov 03 21:24:14.784472 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.784424 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istio-podinfo\") pod \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " Nov 03 21:24:14.784650 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.784491 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istio-envoy\") pod \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " Nov 03 21:24:14.784650 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.784567 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-workload-socket\") pod \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " Nov 03 21:24:14.784650 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.784633 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-workload-certs\") pod \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " Nov 03 21:24:14.784806 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.784665 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-credential-socket" pod "c34e4dc8-c666-4cf5-9a18-7fafb9607383" (UID: "c34e4dc8-c666-4cf5-9a18-7fafb9607383"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:24:14.784806 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.784705 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istiod-ca-cert\") pod \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " Nov 03 21:24:14.784806 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.784721 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istio-data" pod "c34e4dc8-c666-4cf5-9a18-7fafb9607383" (UID: "c34e4dc8-c666-4cf5-9a18-7fafb9607383"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:24:14.784957 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.784825 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c34e4dc8-c666-4cf5-9a18-7fafb9607383-kube-api-access-fr7gc\" (UniqueName: \"kubernetes.io/projected/c34e4dc8-c666-4cf5-9a18-7fafb9607383-kube-api-access-fr7gc\") pod \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\" (UID: \"c34e4dc8-c666-4cf5-9a18-7fafb9607383\") " Nov 03 21:24:14.784957 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.784882 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-workload-socket" pod "c34e4dc8-c666-4cf5-9a18-7fafb9607383" (UID: "c34e4dc8-c666-4cf5-9a18-7fafb9607383"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:24:14.784957 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.784903 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-workload-certs" pod "c34e4dc8-c666-4cf5-9a18-7fafb9607383" (UID: "c34e4dc8-c666-4cf5-9a18-7fafb9607383"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:24:14.785151 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.785054 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istiod-ca-cert" pod "c34e4dc8-c666-4cf5-9a18-7fafb9607383" (UID: "c34e4dc8-c666-4cf5-9a18-7fafb9607383"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:24:14.785431 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.785406 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:14.785431 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.785434 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:14.785642 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.785448 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:14.785642 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.785462 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:14.785642 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.785476 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:14.786688 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.786668 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istio-token" pod "c34e4dc8-c666-4cf5-9a18-7fafb9607383" (UID: "c34e4dc8-c666-4cf5-9a18-7fafb9607383"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:24:14.786851 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.786836 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istio-podinfo" pod "c34e4dc8-c666-4cf5-9a18-7fafb9607383" (UID: "c34e4dc8-c666-4cf5-9a18-7fafb9607383"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:24:14.787041 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.787025 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istio-envoy" pod "c34e4dc8-c666-4cf5-9a18-7fafb9607383" (UID: "c34e4dc8-c666-4cf5-9a18-7fafb9607383"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:24:14.787122 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.787097 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34e4dc8-c666-4cf5-9a18-7fafb9607383-kube-api-access-fr7gc" pod "c34e4dc8-c666-4cf5-9a18-7fafb9607383" (UID: "c34e4dc8-c666-4cf5-9a18-7fafb9607383"). InnerVolumeSpecName "kube-api-access-fr7gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:24:14.820826 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.820801 5615 scope.go:122] "RemoveContainer" containerID="d85831cf86d0d2f19fbb138b983e13c980a884854f41b19c840e2503d10c6de2" Nov 03 21:24:14.821243 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:24:14.821207 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d85831cf86d0d2f19fbb138b983e13c980a884854f41b19c840e2503d10c6de2\": container with ID starting with d85831cf86d0d2f19fbb138b983e13c980a884854f41b19c840e2503d10c6de2 not found: ID does not exist" containerID="d85831cf86d0d2f19fbb138b983e13c980a884854f41b19c840e2503d10c6de2" Nov 03 21:24:14.821426 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.821249 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d85831cf86d0d2f19fbb138b983e13c980a884854f41b19c840e2503d10c6de2"} err="failed to get container status \"d85831cf86d0d2f19fbb138b983e13c980a884854f41b19c840e2503d10c6de2\": rpc error: code = NotFound desc = could not find container \"d85831cf86d0d2f19fbb138b983e13c980a884854f41b19c840e2503d10c6de2\": container with ID starting with d85831cf86d0d2f19fbb138b983e13c980a884854f41b19c840e2503d10c6de2 not found: ID does not exist" Nov 03 21:24:14.887521 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.887475 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:14.887521 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.887522 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:14.887726 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.887538 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c34e4dc8-c666-4cf5-9a18-7fafb9607383-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:14.887726 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.887553 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr7gc\" (UniqueName: \"kubernetes.io/projected/c34e4dc8-c666-4cf5-9a18-7fafb9607383-kube-api-access-fr7gc\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:14.979794 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:14.979767 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:24:15.090785 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.090744 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-workload-certs\") pod \"f5382faa-4094-4acf-a7e8-4538976acfb8\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " Nov 03 21:24:15.090991 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.090850 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5382faa-4094-4acf-a7e8-4538976acfb8-kube-api-access-9t76h\" (UniqueName: \"kubernetes.io/projected/f5382faa-4094-4acf-a7e8-4538976acfb8-kube-api-access-9t76h\") pod \"f5382faa-4094-4acf-a7e8-4538976acfb8\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " Nov 03 21:24:15.090991 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.090914 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-workload-socket\") pod \"f5382faa-4094-4acf-a7e8-4538976acfb8\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " Nov 03 21:24:15.090991 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.090936 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-istio-envoy\") pod \"f5382faa-4094-4acf-a7e8-4538976acfb8\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " Nov 03 21:24:15.090991 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.090975 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5382faa-4094-4acf-a7e8-4538976acfb8-istio-token\" (UniqueName: \"kubernetes.io/projected/f5382faa-4094-4acf-a7e8-4538976acfb8-istio-token\") pod \"f5382faa-4094-4acf-a7e8-4538976acfb8\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " Nov 03 21:24:15.091221 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.091037 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-workload-certs" pod "f5382faa-4094-4acf-a7e8-4538976acfb8" (UID: "f5382faa-4094-4acf-a7e8-4538976acfb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:24:15.091221 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.091063 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5382faa-4094-4acf-a7e8-4538976acfb8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5382faa-4094-4acf-a7e8-4538976acfb8-istio-podinfo\") pod \"f5382faa-4094-4acf-a7e8-4538976acfb8\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " Nov 03 21:24:15.091221 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.091119 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-credential-socket\") pod \"f5382faa-4094-4acf-a7e8-4538976acfb8\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " Nov 03 21:24:15.091221 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.091173 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5382faa-4094-4acf-a7e8-4538976acfb8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5382faa-4094-4acf-a7e8-4538976acfb8-istiod-ca-cert\") pod \"f5382faa-4094-4acf-a7e8-4538976acfb8\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " Nov 03 21:24:15.091423 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.091213 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-workload-socket" pod "f5382faa-4094-4acf-a7e8-4538976acfb8" (UID: "f5382faa-4094-4acf-a7e8-4538976acfb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:24:15.091423 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.091270 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-istio-data\") pod \"f5382faa-4094-4acf-a7e8-4538976acfb8\" (UID: \"f5382faa-4094-4acf-a7e8-4538976acfb8\") " Nov 03 21:24:15.091532 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.091451 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-credential-socket" pod "f5382faa-4094-4acf-a7e8-4538976acfb8" (UID: "f5382faa-4094-4acf-a7e8-4538976acfb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:24:15.091638 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.091618 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-istio-data" pod "f5382faa-4094-4acf-a7e8-4538976acfb8" (UID: "f5382faa-4094-4acf-a7e8-4538976acfb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:24:15.091714 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.091648 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5382faa-4094-4acf-a7e8-4538976acfb8-istiod-ca-cert" pod "f5382faa-4094-4acf-a7e8-4538976acfb8" (UID: "f5382faa-4094-4acf-a7e8-4538976acfb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:24:15.091930 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.091906 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:15.092003 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.091934 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:15.092003 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.091950 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5382faa-4094-4acf-a7e8-4538976acfb8-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:15.092003 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.091966 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:15.092003 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.091981 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:15.093480 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.093460 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-istio-envoy" pod "f5382faa-4094-4acf-a7e8-4538976acfb8" (UID: "f5382faa-4094-4acf-a7e8-4538976acfb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:24:15.093561 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.093488 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5382faa-4094-4acf-a7e8-4538976acfb8-kube-api-access-9t76h" pod "f5382faa-4094-4acf-a7e8-4538976acfb8" (UID: "f5382faa-4094-4acf-a7e8-4538976acfb8"). InnerVolumeSpecName "kube-api-access-9t76h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:24:15.093648 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.093629 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5382faa-4094-4acf-a7e8-4538976acfb8-istio-podinfo" pod "f5382faa-4094-4acf-a7e8-4538976acfb8" (UID: "f5382faa-4094-4acf-a7e8-4538976acfb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:24:15.093715 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.093621 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5382faa-4094-4acf-a7e8-4538976acfb8-istio-token" pod "f5382faa-4094-4acf-a7e8-4538976acfb8" (UID: "f5382faa-4094-4acf-a7e8-4538976acfb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:24:15.193379 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.193336 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5382faa-4094-4acf-a7e8-4538976acfb8-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:15.193379 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.193374 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5382faa-4094-4acf-a7e8-4538976acfb8-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:15.193379 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.193383 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5382faa-4094-4acf-a7e8-4538976acfb8-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:15.193644 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.193393 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t76h\" (UniqueName: \"kubernetes.io/projected/f5382faa-4094-4acf-a7e8-4538976acfb8-kube-api-access-9t76h\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:15.202454 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.202395 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk"] Nov 03 21:24:15.206669 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.206632 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-vz4dk"] Nov 03 21:24:15.652325 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.652290 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c34e4dc8-c666-4cf5-9a18-7fafb9607383" path="/var/lib/kubelet/pods/c34e4dc8-c666-4cf5-9a18-7fafb9607383/volumes" Nov 03 21:24:15.743619 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.743582 5615 generic.go:356] "Generic (PLEG): container finished" podID="f5382faa-4094-4acf-a7e8-4538976acfb8" containerID="7a7af3bae7574fe7c19d468a021dc7bed857c50238383024db72ba5990a8ba80" exitCode=0 Nov 03 21:24:15.744122 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.743631 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" Nov 03 21:24:15.744122 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.743659 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" event={"ID":"f5382faa-4094-4acf-a7e8-4538976acfb8","Type":"ContainerDied","Data":"7a7af3bae7574fe7c19d468a021dc7bed857c50238383024db72ba5990a8ba80"} Nov 03 21:24:15.744122 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.743706 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh" event={"ID":"f5382faa-4094-4acf-a7e8-4538976acfb8","Type":"ContainerDied","Data":"4d2a7dc191b09887b78fc5549b143c5dee6405a47cfeeb33afdb1e09bc1b0d53"} Nov 03 21:24:15.744122 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.743737 5615 scope.go:122] "RemoveContainer" containerID="7a7af3bae7574fe7c19d468a021dc7bed857c50238383024db72ba5990a8ba80" Nov 03 21:24:15.753318 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.753296 5615 scope.go:122] "RemoveContainer" containerID="7a7af3bae7574fe7c19d468a021dc7bed857c50238383024db72ba5990a8ba80" Nov 03 21:24:15.753606 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:24:15.753585 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a7af3bae7574fe7c19d468a021dc7bed857c50238383024db72ba5990a8ba80\": container with ID starting with 7a7af3bae7574fe7c19d468a021dc7bed857c50238383024db72ba5990a8ba80 not found: ID does not exist" containerID="7a7af3bae7574fe7c19d468a021dc7bed857c50238383024db72ba5990a8ba80" Nov 03 21:24:15.753658 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.753615 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a7af3bae7574fe7c19d468a021dc7bed857c50238383024db72ba5990a8ba80"} err="failed to get container status \"7a7af3bae7574fe7c19d468a021dc7bed857c50238383024db72ba5990a8ba80\": rpc error: code = NotFound desc = could not find container \"7a7af3bae7574fe7c19d468a021dc7bed857c50238383024db72ba5990a8ba80\": container with ID starting with 7a7af3bae7574fe7c19d468a021dc7bed857c50238383024db72ba5990a8ba80 not found: ID does not exist" Nov 03 21:24:15.901474 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.901426 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh"] Nov 03 21:24:15.948693 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:15.948654 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m24hh"] Nov 03 21:24:17.536566 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:17.536529 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d"] Nov 03 21:24:17.537187 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:17.537172 5615 cpu_manager.go:414] "containerMap: removing container" podUID="d22b9636-a144-4ba6-b538-d22b253c8908" containerName="maas-api" Nov 03 21:24:17.537239 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:17.537187 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d22b9636-a144-4ba6-b538-d22b253c8908" containerName="maas-api" Nov 03 21:24:17.537239 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:17.537201 5615 cpu_manager.go:414] "containerMap: removing container" podUID="c34e4dc8-c666-4cf5-9a18-7fafb9607383" containerName="istio-proxy" Nov 03 21:24:17.537239 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:17.537207 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34e4dc8-c666-4cf5-9a18-7fafb9607383" containerName="istio-proxy" Nov 03 21:24:17.537239 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:17.537218 5615 cpu_manager.go:414] "containerMap: removing container" podUID="f5382faa-4094-4acf-a7e8-4538976acfb8" containerName="istio-proxy" Nov 03 21:24:17.537239 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:17.537223 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5382faa-4094-4acf-a7e8-4538976acfb8" containerName="istio-proxy" Nov 03 21:24:17.537393 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:17.537299 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="d22b9636-a144-4ba6-b538-d22b253c8908" containerName="maas-api" Nov 03 21:24:17.537393 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:17.537309 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34e4dc8-c666-4cf5-9a18-7fafb9607383" containerName="istio-proxy" Nov 03 21:24:17.537393 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:17.537317 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5382faa-4094-4acf-a7e8-4538976acfb8" containerName="istio-proxy" Nov 03 21:24:17.542168 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:17.542145 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d" Nov 03 21:24:17.593466 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:17.593436 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-1dda2c-serving-cert\"" Nov 03 21:24:17.624509 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:17.624459 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/342a2378-e673-42df-bc30-91af7ba1c534-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d\" (UID: \"342a2378-e673-42df-bc30-91af7ba1c534\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d" Nov 03 21:24:17.624685 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:17.624519 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/342a2378-e673-42df-bc30-91af7ba1c534-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d\" (UID: \"342a2378-e673-42df-bc30-91af7ba1c534\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d" Nov 03 21:24:17.624750 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:17.624681 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8xjl\" (UniqueName: \"kubernetes.io/projected/342a2378-e673-42df-bc30-91af7ba1c534-kube-api-access-v8xjl\") pod \"maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d\" (UID: \"342a2378-e673-42df-bc30-91af7ba1c534\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d" Nov 03 21:24:17.624860 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:17.624841 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/342a2378-e673-42df-bc30-91af7ba1c534-metrics-certs\") pod \"maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d\" (UID: \"342a2378-e673-42df-bc30-91af7ba1c534\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d" Nov 03 21:24:17.643338 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:17.643302 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d"] Nov 03 21:24:17.651840 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:17.651809 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5382faa-4094-4acf-a7e8-4538976acfb8" path="/var/lib/kubelet/pods/f5382faa-4094-4acf-a7e8-4538976acfb8/volumes" Nov 03 21:24:17.726441 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:17.726400 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/342a2378-e673-42df-bc30-91af7ba1c534-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d\" (UID: \"342a2378-e673-42df-bc30-91af7ba1c534\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d" Nov 03 21:24:17.726620 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:17.726451 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/342a2378-e673-42df-bc30-91af7ba1c534-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d\" (UID: \"342a2378-e673-42df-bc30-91af7ba1c534\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d" Nov 03 21:24:17.726620 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:17.726487 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8xjl\" (UniqueName: \"kubernetes.io/projected/342a2378-e673-42df-bc30-91af7ba1c534-kube-api-access-v8xjl\") pod \"maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d\" (UID: \"342a2378-e673-42df-bc30-91af7ba1c534\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d" Nov 03 21:24:17.726620 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:24:17.726551 5615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-serving-cert: secret "maas-api-e2e-praxis-b-1dda2c-serving-cert" not found Nov 03 21:24:17.726620 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:24:17.726619 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/342a2378-e673-42df-bc30-91af7ba1c534-maas-api-tls podName:342a2378-e673-42df-bc30-91af7ba1c534 nodeName:}" failed. No retries permitted until 2026-11-03 21:24:18.226599654 +0000 UTC m=+2152.694378356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/342a2378-e673-42df-bc30-91af7ba1c534-maas-api-tls") pod "maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d" (UID: "342a2378-e673-42df-bc30-91af7ba1c534") : secret "maas-api-e2e-praxis-b-1dda2c-serving-cert" not found Nov 03 21:24:17.726897 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:17.726713 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/342a2378-e673-42df-bc30-91af7ba1c534-metrics-certs\") pod \"maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d\" (UID: \"342a2378-e673-42df-bc30-91af7ba1c534\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d" Nov 03 21:24:17.727275 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:17.727255 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/342a2378-e673-42df-bc30-91af7ba1c534-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d\" (UID: \"342a2378-e673-42df-bc30-91af7ba1c534\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d" Nov 03 21:24:17.729159 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:17.729138 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/342a2378-e673-42df-bc30-91af7ba1c534-metrics-certs\") pod \"maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d\" (UID: \"342a2378-e673-42df-bc30-91af7ba1c534\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d" Nov 03 21:24:17.785048 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:17.785021 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8xjl\" (UniqueName: \"kubernetes.io/projected/342a2378-e673-42df-bc30-91af7ba1c534-kube-api-access-v8xjl\") pod \"maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d\" (UID: \"342a2378-e673-42df-bc30-91af7ba1c534\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d" Nov 03 21:24:18.234850 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:18.234810 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/342a2378-e673-42df-bc30-91af7ba1c534-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d\" (UID: \"342a2378-e673-42df-bc30-91af7ba1c534\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d" Nov 03 21:24:18.237364 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:18.237342 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/342a2378-e673-42df-bc30-91af7ba1c534-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d\" (UID: \"342a2378-e673-42df-bc30-91af7ba1c534\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d" Nov 03 21:24:18.455489 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:18.455454 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d" Nov 03 21:24:18.836030 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:18.836002 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d"] Nov 03 21:24:18.838370 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:24:18.838333 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod342a2378_e673_42df_bc30_91af7ba1c534.slice/crio-2e3297f6a4b9acb12a2c6505818f2e9df53cde7a5eaee77d36dd75640a95863b WatchSource:0}: Error finding container 2e3297f6a4b9acb12a2c6505818f2e9df53cde7a5eaee77d36dd75640a95863b: Status 404 returned error can't find the container with id 2e3297f6a4b9acb12a2c6505818f2e9df53cde7a5eaee77d36dd75640a95863b Nov 03 21:24:19.771116 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:19.771083 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d" event={"ID":"342a2378-e673-42df-bc30-91af7ba1c534","Type":"ContainerStarted","Data":"2e3297f6a4b9acb12a2c6505818f2e9df53cde7a5eaee77d36dd75640a95863b"} Nov 03 21:24:20.778774 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:20.778733 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d" event={"ID":"342a2378-e673-42df-bc30-91af7ba1c534","Type":"ContainerStarted","Data":"29e687e739ace9ba589b2e71fe10b2953eabb7f2c63056827d5357ef263c9cb9"} Nov 03 21:24:20.779270 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:20.778811 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d" Nov 03 21:24:20.891304 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:20.891253 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d" podStartSLOduration=2.962467267 podStartE2EDuration="3.891238977s" totalImagesPullingTime="928.77171ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:24:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:24:20.890482885 +0000 UTC m=+2155.358261608" watchObservedRunningTime="2026-11-03 21:24:20.891238977 +0000 UTC m=+2155.359017700" Nov 03 21:24:26.790470 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:26.790437 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d" Nov 03 21:24:40.619973 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:24:40.619896 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:24:49.633006 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.632965 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw"] Nov 03 21:24:49.637397 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.637368 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:49.716371 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.716327 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istio-podinfo\") pod \"e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:49.716537 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.716497 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istio-token\") pod \"e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:49.716579 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.716548 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-credential-socket\") pod \"e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:49.716614 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.716580 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-workload-socket\") pod \"e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:49.716701 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.716680 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istio-data\") pod \"e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:49.716739 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.716710 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istio-envoy\") pod \"e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:49.716784 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.716766 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istiod-ca-cert\") pod \"e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:49.716833 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.716821 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4jl7\" (UniqueName: \"kubernetes.io/projected/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-kube-api-access-k4jl7\") pod \"e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:49.716895 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.716881 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-workload-certs\") pod \"e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:49.733775 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.733739 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw"] Nov 03 21:24:49.745177 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.745140 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-fe2df9-openshift-default-dockercfg-cj86k\"" Nov 03 21:24:49.818353 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.818306 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istio-token\") pod \"e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:49.818571 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.818371 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-credential-socket\") pod \"e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:49.818571 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.818404 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-workload-socket\") pod \"e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:49.818571 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.818473 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istio-data\") pod \"e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:49.818571 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.818501 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istio-envoy\") pod \"e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:49.818571 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.818529 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istiod-ca-cert\") pod \"e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:49.818882 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.818604 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4jl7\" (UniqueName: \"kubernetes.io/projected/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-kube-api-access-k4jl7\") pod \"e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:49.818882 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.818640 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-workload-certs\") pod \"e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:49.818882 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.818753 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istio-podinfo\") pod \"e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:49.821951 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.820652 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istio-data\") pod \"e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:49.822307 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.822271 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istiod-ca-cert\") pod \"e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:49.822928 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.822894 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-credential-socket\") pod \"e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:49.823183 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.823159 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-workload-socket\") pod \"e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:49.823964 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.823936 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-workload-certs\") pod \"e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:49.826253 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.826167 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istio-envoy\") pod \"e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:49.826557 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.826465 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istio-podinfo\") pod \"e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:49.880962 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.880925 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istio-token\") pod \"e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:49.881154 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.881117 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4jl7\" (UniqueName: \"kubernetes.io/projected/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-kube-api-access-k4jl7\") pod \"e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:49.952394 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:49.952307 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:50.400635 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:50.400608 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw"] Nov 03 21:24:50.402315 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:24:50.402272 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a9e3ce_28e8_4a93_a6aa_ad2073cb4b87.slice/crio-d0ba26a9c2dd6ab9d0558c8c602822a7cf69de133ee82fdb01b0735793322783 WatchSource:0}: Error finding container d0ba26a9c2dd6ab9d0558c8c602822a7cf69de133ee82fdb01b0735793322783: Status 404 returned error can't find the container with id d0ba26a9c2dd6ab9d0558c8c602822a7cf69de133ee82fdb01b0735793322783 Nov 03 21:24:50.404964 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:50.404930 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:24:50.405042 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:50.404999 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:24:50.405042 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:50.405030 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:24:50.968177 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:50.968134 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" event={"ID":"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87","Type":"ContainerStarted","Data":"37c59b60e2c16777df037693df6ddad73ffb754cd04ed153ec359cd37fcf4ee0"} Nov 03 21:24:50.968634 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:50.968191 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" event={"ID":"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87","Type":"ContainerStarted","Data":"d0ba26a9c2dd6ab9d0558c8c602822a7cf69de133ee82fdb01b0735793322783"} Nov 03 21:24:51.085946 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:51.085890 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" podStartSLOduration=2.085874903 podStartE2EDuration="2.085874903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:24:51.083855423 +0000 UTC m=+2185.551634147" watchObservedRunningTime="2026-11-03 21:24:51.085874903 +0000 UTC m=+2185.553653626" Nov 03 21:24:51.654104 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:51.654055 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw"] Nov 03 21:24:51.658548 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:51.658521 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw" Nov 03 21:24:51.709119 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:51.709065 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-b5c926-serving-cert\"" Nov 03 21:24:51.747343 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:51.747287 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ce14fb0-6ee8-441d-ac37-4207993f445d-metrics-certs\") pod \"maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw\" (UID: \"3ce14fb0-6ee8-441d-ac37-4207993f445d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw" Nov 03 21:24:51.747590 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:51.747564 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ce14fb0-6ee8-441d-ac37-4207993f445d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw\" (UID: \"3ce14fb0-6ee8-441d-ac37-4207993f445d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw" Nov 03 21:24:51.747815 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:51.747789 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxfxk\" (UniqueName: \"kubernetes.io/projected/3ce14fb0-6ee8-441d-ac37-4207993f445d-kube-api-access-kxfxk\") pod \"maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw\" (UID: \"3ce14fb0-6ee8-441d-ac37-4207993f445d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw" Nov 03 21:24:51.747891 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:51.747832 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ce14fb0-6ee8-441d-ac37-4207993f445d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw\" (UID: \"3ce14fb0-6ee8-441d-ac37-4207993f445d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw" Nov 03 21:24:51.759042 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:51.759011 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw"] Nov 03 21:24:51.849385 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:51.849338 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxfxk\" (UniqueName: \"kubernetes.io/projected/3ce14fb0-6ee8-441d-ac37-4207993f445d-kube-api-access-kxfxk\") pod \"maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw\" (UID: \"3ce14fb0-6ee8-441d-ac37-4207993f445d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw" Nov 03 21:24:51.849385 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:51.849388 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ce14fb0-6ee8-441d-ac37-4207993f445d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw\" (UID: \"3ce14fb0-6ee8-441d-ac37-4207993f445d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw" Nov 03 21:24:51.849628 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:51.849472 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ce14fb0-6ee8-441d-ac37-4207993f445d-metrics-certs\") pod \"maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw\" (UID: \"3ce14fb0-6ee8-441d-ac37-4207993f445d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw" Nov 03 21:24:51.849628 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:51.849548 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ce14fb0-6ee8-441d-ac37-4207993f445d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw\" (UID: \"3ce14fb0-6ee8-441d-ac37-4207993f445d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw" Nov 03 21:24:51.849628 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:24:51.849562 5615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-serving-cert: secret "maas-api-e2e-praxis-a-b5c926-serving-cert" not found Nov 03 21:24:51.849748 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:24:51.849631 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ce14fb0-6ee8-441d-ac37-4207993f445d-maas-api-tls podName:3ce14fb0-6ee8-441d-ac37-4207993f445d nodeName:}" failed. No retries permitted until 2026-11-03 21:24:52.349609804 +0000 UTC m=+2186.817388508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3ce14fb0-6ee8-441d-ac37-4207993f445d-maas-api-tls") pod "maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw" (UID: "3ce14fb0-6ee8-441d-ac37-4207993f445d") : secret "maas-api-e2e-praxis-a-b5c926-serving-cert" not found Nov 03 21:24:51.850306 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:51.850280 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ce14fb0-6ee8-441d-ac37-4207993f445d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw\" (UID: \"3ce14fb0-6ee8-441d-ac37-4207993f445d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw" Nov 03 21:24:51.852124 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:51.852100 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ce14fb0-6ee8-441d-ac37-4207993f445d-metrics-certs\") pod \"maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw\" (UID: \"3ce14fb0-6ee8-441d-ac37-4207993f445d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw" Nov 03 21:24:51.935147 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:51.934641 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxfxk\" (UniqueName: \"kubernetes.io/projected/3ce14fb0-6ee8-441d-ac37-4207993f445d-kube-api-access-kxfxk\") pod \"maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw\" (UID: \"3ce14fb0-6ee8-441d-ac37-4207993f445d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw" Nov 03 21:24:51.952927 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:51.952893 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:51.959204 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:51.959175 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:51.975992 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:51.975957 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:51.976997 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:51.976975 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:24:52.361853 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:52.361810 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ce14fb0-6ee8-441d-ac37-4207993f445d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw\" (UID: \"3ce14fb0-6ee8-441d-ac37-4207993f445d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw" Nov 03 21:24:52.364279 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:52.364254 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ce14fb0-6ee8-441d-ac37-4207993f445d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw\" (UID: \"3ce14fb0-6ee8-441d-ac37-4207993f445d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw" Nov 03 21:24:52.573384 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:52.573352 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw" Nov 03 21:24:52.947610 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:52.947578 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw"] Nov 03 21:24:52.949229 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:24:52.949185 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce14fb0_6ee8_441d_ac37_4207993f445d.slice/crio-d230cf3a7d6866b8e45e93e2148f249ac258a8cd3219a149f808e118b57b634f WatchSource:0}: Error finding container d230cf3a7d6866b8e45e93e2148f249ac258a8cd3219a149f808e118b57b634f: Status 404 returned error can't find the container with id d230cf3a7d6866b8e45e93e2148f249ac258a8cd3219a149f808e118b57b634f Nov 03 21:24:52.983966 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:52.983931 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw" event={"ID":"3ce14fb0-6ee8-441d-ac37-4207993f445d","Type":"ContainerStarted","Data":"d230cf3a7d6866b8e45e93e2148f249ac258a8cd3219a149f808e118b57b634f"} Nov 03 21:24:53.991099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:53.991042 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw" event={"ID":"3ce14fb0-6ee8-441d-ac37-4207993f445d","Type":"ContainerStarted","Data":"ae3017a9e104144620da8f1097df4c7d5a309a087d0a4bac84b1d48efbb5abae"} Nov 03 21:24:53.991550 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:53.991146 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw" Nov 03 21:24:54.144839 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:24:54.144713 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw" podStartSLOduration=2.237983913 podStartE2EDuration="3.144697313s" totalImagesPullingTime="906.7134ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:24:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:24:54.143467684 +0000 UTC m=+2188.611246408" watchObservedRunningTime="2026-11-03 21:24:54.144697313 +0000 UTC m=+2188.612476059" Nov 03 21:25:00.003684 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:00.003656 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw" Nov 03 21:25:01.031678 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:01.031637 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t"] Nov 03 21:25:01.032134 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:01.031855 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t" podUID="314a2820-1a0c-477b-adab-a2428c552b65" containerName="maas-api" containerID="cri-o://47a0fd4dadc6a6146affe1a6ca85ea9f527a7b33b3ab46ad1b640d812876deb8" gracePeriod=30 Nov 03 21:25:01.342407 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:01.342374 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t" Nov 03 21:25:01.482697 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:01.482658 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/314a2820-1a0c-477b-adab-a2428c552b65-kube-api-access-r6rs6\" (UniqueName: \"kubernetes.io/projected/314a2820-1a0c-477b-adab-a2428c552b65-kube-api-access-r6rs6\") pod \"314a2820-1a0c-477b-adab-a2428c552b65\" (UID: \"314a2820-1a0c-477b-adab-a2428c552b65\") " Nov 03 21:25:01.482908 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:01.482734 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/314a2820-1a0c-477b-adab-a2428c552b65-metrics-certs\" (UniqueName: \"kubernetes.io/secret/314a2820-1a0c-477b-adab-a2428c552b65-metrics-certs\") pod \"314a2820-1a0c-477b-adab-a2428c552b65\" (UID: \"314a2820-1a0c-477b-adab-a2428c552b65\") " Nov 03 21:25:01.482908 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:01.482776 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/314a2820-1a0c-477b-adab-a2428c552b65-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/314a2820-1a0c-477b-adab-a2428c552b65-maas-api-tls\") pod \"314a2820-1a0c-477b-adab-a2428c552b65\" (UID: \"314a2820-1a0c-477b-adab-a2428c552b65\") " Nov 03 21:25:01.483165 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:01.482929 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/314a2820-1a0c-477b-adab-a2428c552b65-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/314a2820-1a0c-477b-adab-a2428c552b65-openshift-service-ca\") pod \"314a2820-1a0c-477b-adab-a2428c552b65\" (UID: \"314a2820-1a0c-477b-adab-a2428c552b65\") " Nov 03 21:25:01.483634 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:01.483597 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/314a2820-1a0c-477b-adab-a2428c552b65-openshift-service-ca" pod "314a2820-1a0c-477b-adab-a2428c552b65" (UID: "314a2820-1a0c-477b-adab-a2428c552b65"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:25:01.485555 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:01.485514 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/314a2820-1a0c-477b-adab-a2428c552b65-metrics-certs" pod "314a2820-1a0c-477b-adab-a2428c552b65" (UID: "314a2820-1a0c-477b-adab-a2428c552b65"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:25:01.485794 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:01.485769 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/314a2820-1a0c-477b-adab-a2428c552b65-kube-api-access-r6rs6" pod "314a2820-1a0c-477b-adab-a2428c552b65" (UID: "314a2820-1a0c-477b-adab-a2428c552b65"). InnerVolumeSpecName "kube-api-access-r6rs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:25:01.486179 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:01.486155 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/314a2820-1a0c-477b-adab-a2428c552b65-maas-api-tls" pod "314a2820-1a0c-477b-adab-a2428c552b65" (UID: "314a2820-1a0c-477b-adab-a2428c552b65"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:25:01.592450 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:01.592289 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6rs6\" (UniqueName: \"kubernetes.io/projected/314a2820-1a0c-477b-adab-a2428c552b65-kube-api-access-r6rs6\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:25:01.592450 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:01.592334 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/314a2820-1a0c-477b-adab-a2428c552b65-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:25:01.592450 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:01.592351 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/314a2820-1a0c-477b-adab-a2428c552b65-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:25:01.592450 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:01.592365 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/314a2820-1a0c-477b-adab-a2428c552b65-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:25:02.080649 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:02.073219 5615 generic.go:356] "Generic (PLEG): container finished" podID="314a2820-1a0c-477b-adab-a2428c552b65" containerID="47a0fd4dadc6a6146affe1a6ca85ea9f527a7b33b3ab46ad1b640d812876deb8" exitCode=0 Nov 03 21:25:02.080649 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:02.073458 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t" event={"ID":"314a2820-1a0c-477b-adab-a2428c552b65","Type":"ContainerDied","Data":"47a0fd4dadc6a6146affe1a6ca85ea9f527a7b33b3ab46ad1b640d812876deb8"} Nov 03 21:25:02.080649 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:02.073515 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t" event={"ID":"314a2820-1a0c-477b-adab-a2428c552b65","Type":"ContainerDied","Data":"01de04916ae7f9fec618980f85689f89644f8f054de128b3c1c8d2629d0689e8"} Nov 03 21:25:02.080649 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:02.073556 5615 scope.go:122] "RemoveContainer" containerID="47a0fd4dadc6a6146affe1a6ca85ea9f527a7b33b3ab46ad1b640d812876deb8" Nov 03 21:25:02.080649 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:02.073730 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t" Nov 03 21:25:02.102295 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:02.102265 5615 scope.go:122] "RemoveContainer" containerID="47a0fd4dadc6a6146affe1a6ca85ea9f527a7b33b3ab46ad1b640d812876deb8" Nov 03 21:25:02.102668 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:25:02.102639 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47a0fd4dadc6a6146affe1a6ca85ea9f527a7b33b3ab46ad1b640d812876deb8\": container with ID starting with 47a0fd4dadc6a6146affe1a6ca85ea9f527a7b33b3ab46ad1b640d812876deb8 not found: ID does not exist" containerID="47a0fd4dadc6a6146affe1a6ca85ea9f527a7b33b3ab46ad1b640d812876deb8" Nov 03 21:25:02.102767 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:02.102679 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47a0fd4dadc6a6146affe1a6ca85ea9f527a7b33b3ab46ad1b640d812876deb8"} err="failed to get container status \"47a0fd4dadc6a6146affe1a6ca85ea9f527a7b33b3ab46ad1b640d812876deb8\": rpc error: code = NotFound desc = could not find container \"47a0fd4dadc6a6146affe1a6ca85ea9f527a7b33b3ab46ad1b640d812876deb8\": container with ID starting with 47a0fd4dadc6a6146affe1a6ca85ea9f527a7b33b3ab46ad1b640d812876deb8 not found: ID does not exist" Nov 03 21:25:02.230398 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:02.230365 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t"] Nov 03 21:25:02.277494 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:02.277454 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7af9dd-6dcf459558-kdp7t"] Nov 03 21:25:03.651623 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:03.651589 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="314a2820-1a0c-477b-adab-a2428c552b65" path="/var/lib/kubelet/pods/314a2820-1a0c-477b-adab-a2428c552b65/volumes" Nov 03 21:25:10.620754 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:25:10.620710 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:25:12.964621 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:12.964583 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5"] Nov 03 21:25:12.965188 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:12.964864 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" podUID="a41dcb03-3288-4425-866e-4473c432b8b5" containerName="istio-proxy" containerID="cri-o://6af3dcef8ada1fd8d46dfcc4c69138b7080a8889d1fa1e5ad988ac2efdf5748f" gracePeriod=30 Nov 03 21:25:15.087777 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:15.087741 5615 patch_prober.go:29] interesting pod/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:25:15.088340 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:15.087805 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" podUID="a41dcb03-3288-4425-866e-4473c432b8b5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:25:15.216632 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:15.216597 5615 generic.go:356] "Generic (PLEG): container finished" podID="09573228-1c0e-478a-b836-7b46b30e6c99" containerID="6a77bdf86d0c69ccda26c60cf9cc5dc9b8646ac2c0c2f83ef912dc04ee6cfef2" exitCode=0 Nov 03 21:25:15.216804 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:15.216671 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" event={"ID":"09573228-1c0e-478a-b836-7b46b30e6c99","Type":"ContainerDied","Data":"6a77bdf86d0c69ccda26c60cf9cc5dc9b8646ac2c0c2f83ef912dc04ee6cfef2"} Nov 03 21:25:15.217089 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:15.217063 5615 scope.go:122] "RemoveContainer" containerID="6a77bdf86d0c69ccda26c60cf9cc5dc9b8646ac2c0c2f83ef912dc04ee6cfef2" Nov 03 21:25:15.218608 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:15.218580 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:25:15.218668 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:15.218655 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:25:15.218706 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:15.218691 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:25:16.228143 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:16.228102 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" event={"ID":"09573228-1c0e-478a-b836-7b46b30e6c99","Type":"ContainerStarted","Data":"c7512935571d1586146460751075aa7feaf08f4b7f9b3176117f520491e0afd1"} Nov 03 21:25:16.788589 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:16.788549 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:25:16.788804 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:16.788717 5615 patch_prober.go:29] interesting pod/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 03 21:25:16.788804 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:16.788779 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" podUID="09573228-1c0e-478a-b836-7b46b30e6c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 03 21:25:17.788659 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:17.788617 5615 patch_prober.go:29] interesting pod/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 03 21:25:17.789186 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:17.788676 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" podUID="09573228-1c0e-478a-b836-7b46b30e6c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 03 21:25:18.247123 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.247092 5615 generic.go:356] "Generic (PLEG): container finished" podID="a41dcb03-3288-4425-866e-4473c432b8b5" containerID="6af3dcef8ada1fd8d46dfcc4c69138b7080a8889d1fa1e5ad988ac2efdf5748f" exitCode=0 Nov 03 21:25:18.247302 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.247284 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" event={"ID":"a41dcb03-3288-4425-866e-4473c432b8b5","Type":"ContainerDied","Data":"6af3dcef8ada1fd8d46dfcc4c69138b7080a8889d1fa1e5ad988ac2efdf5748f"} Nov 03 21:25:18.247382 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.247346 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" event={"ID":"a41dcb03-3288-4425-866e-4473c432b8b5","Type":"ContainerDied","Data":"dffadcad6356ef3ac430317a73a9108f7e71ff350ea6988670cfa080b2c3f9eb"} Nov 03 21:25:18.247382 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.247369 5615 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dffadcad6356ef3ac430317a73a9108f7e71ff350ea6988670cfa080b2c3f9eb" Nov 03 21:25:18.247382 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.247383 5615 scope.go:122] "RemoveContainer" containerID="cd8ea78210960b3e4dfab1f2e7e96e9e26e154688aa823643a10a0e440f71893" Nov 03 21:25:18.268397 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.268241 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:25:18.362940 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.362826 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a41dcb03-3288-4425-866e-4473c432b8b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a41dcb03-3288-4425-866e-4473c432b8b5-istiod-ca-cert\") pod \"a41dcb03-3288-4425-866e-4473c432b8b5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " Nov 03 21:25:18.363167 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.362962 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a41dcb03-3288-4425-866e-4473c432b8b5-istio-token\" (UniqueName: \"kubernetes.io/projected/a41dcb03-3288-4425-866e-4473c432b8b5-istio-token\") pod \"a41dcb03-3288-4425-866e-4473c432b8b5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " Nov 03 21:25:18.363167 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.363003 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-credential-socket\") pod \"a41dcb03-3288-4425-866e-4473c432b8b5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " Nov 03 21:25:18.363167 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.363034 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-workload-socket\") pod \"a41dcb03-3288-4425-866e-4473c432b8b5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " Nov 03 21:25:18.363167 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.363112 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a41dcb03-3288-4425-866e-4473c432b8b5-kube-api-access-p5pz4\" (UniqueName: \"kubernetes.io/projected/a41dcb03-3288-4425-866e-4473c432b8b5-kube-api-access-p5pz4\") pod \"a41dcb03-3288-4425-866e-4473c432b8b5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " Nov 03 21:25:18.363392 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.363245 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-workload-certs\") pod \"a41dcb03-3288-4425-866e-4473c432b8b5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " Nov 03 21:25:18.363392 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.363381 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-istio-data\") pod \"a41dcb03-3288-4425-866e-4473c432b8b5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " Nov 03 21:25:18.363493 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.363417 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a41dcb03-3288-4425-866e-4473c432b8b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a41dcb03-3288-4425-866e-4473c432b8b5-istio-podinfo\") pod \"a41dcb03-3288-4425-866e-4473c432b8b5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " Nov 03 21:25:18.363493 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.363465 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-istio-envoy\") pod \"a41dcb03-3288-4425-866e-4473c432b8b5\" (UID: \"a41dcb03-3288-4425-866e-4473c432b8b5\") " Nov 03 21:25:18.365002 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.364718 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-workload-socket" pod "a41dcb03-3288-4425-866e-4473c432b8b5" (UID: "a41dcb03-3288-4425-866e-4473c432b8b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:25:18.365650 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.365327 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a41dcb03-3288-4425-866e-4473c432b8b5-istiod-ca-cert" pod "a41dcb03-3288-4425-866e-4473c432b8b5" (UID: "a41dcb03-3288-4425-866e-4473c432b8b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:25:18.365650 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.365606 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-istio-data" pod "a41dcb03-3288-4425-866e-4473c432b8b5" (UID: "a41dcb03-3288-4425-866e-4473c432b8b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:25:18.365964 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.365912 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-credential-socket" pod "a41dcb03-3288-4425-866e-4473c432b8b5" (UID: "a41dcb03-3288-4425-866e-4473c432b8b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:25:18.366486 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.366464 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-workload-certs" pod "a41dcb03-3288-4425-866e-4473c432b8b5" (UID: "a41dcb03-3288-4425-866e-4473c432b8b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:25:18.367707 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.367670 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-istio-envoy" pod "a41dcb03-3288-4425-866e-4473c432b8b5" (UID: "a41dcb03-3288-4425-866e-4473c432b8b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:25:18.368647 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.368614 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a41dcb03-3288-4425-866e-4473c432b8b5-istio-token" pod "a41dcb03-3288-4425-866e-4473c432b8b5" (UID: "a41dcb03-3288-4425-866e-4473c432b8b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:25:18.368937 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.368911 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a41dcb03-3288-4425-866e-4473c432b8b5-istio-podinfo" pod "a41dcb03-3288-4425-866e-4473c432b8b5" (UID: "a41dcb03-3288-4425-866e-4473c432b8b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:25:18.369259 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.369223 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a41dcb03-3288-4425-866e-4473c432b8b5-kube-api-access-p5pz4" pod "a41dcb03-3288-4425-866e-4473c432b8b5" (UID: "a41dcb03-3288-4425-866e-4473c432b8b5"). InnerVolumeSpecName "kube-api-access-p5pz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:25:18.465716 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.465672 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:25:18.465716 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.465715 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a41dcb03-3288-4425-866e-4473c432b8b5-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:25:18.465976 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.465732 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:25:18.465976 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.465746 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a41dcb03-3288-4425-866e-4473c432b8b5-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:25:18.465976 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.465760 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a41dcb03-3288-4425-866e-4473c432b8b5-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:25:18.465976 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.465774 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:25:18.465976 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.465788 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:25:18.465976 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.465801 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5pz4\" (UniqueName: \"kubernetes.io/projected/a41dcb03-3288-4425-866e-4473c432b8b5-kube-api-access-p5pz4\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:25:18.465976 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.465814 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a41dcb03-3288-4425-866e-4473c432b8b5-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:25:18.788855 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.788817 5615 patch_prober.go:29] interesting pod/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 03 21:25:18.789274 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:18.788872 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" podUID="09573228-1c0e-478a-b836-7b46b30e6c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 03 21:25:19.255185 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:19.255150 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5" Nov 03 21:25:19.526621 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:19.526530 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5"] Nov 03 21:25:19.531345 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:19.531310 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-7af9dd-openshift-default-775d5d9c85-dvtq5"] Nov 03 21:25:19.653324 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:19.653290 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a41dcb03-3288-4425-866e-4473c432b8b5" path="/var/lib/kubelet/pods/a41dcb03-3288-4425-866e-4473c432b8b5/volumes" Nov 03 21:25:19.788811 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:19.788709 5615 patch_prober.go:29] interesting pod/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 03 21:25:19.788811 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:19.788777 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" podUID="09573228-1c0e-478a-b836-7b46b30e6c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 03 21:25:20.789366 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:20.789331 5615 patch_prober.go:29] interesting pod/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 03 21:25:20.789793 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:20.789391 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" podUID="09573228-1c0e-478a-b836-7b46b30e6c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 03 21:25:21.500368 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:21.500337 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:25:21.793388 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:21.793299 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:25:22.274970 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:22.274937 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:25:40.618978 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:25:40.618933 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:25:43.549971 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:43.549935 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk_ab30a7f6-78c8-4365-b2d5-7040cac18697/payload-processing/0.log" Nov 03 21:25:43.844449 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:43.844363 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-r5zp7_40e8212c-49a4-4a50-add6-0ee8a55429b2/payload-processing/0.log" Nov 03 21:25:44.112138 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:44.112034 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-r5zp7_40e8212c-49a4-4a50-add6-0ee8a55429b2/payload-processing/0.log" Nov 03 21:25:44.340581 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:44.340552 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj_cd915ca2-fb6f-4a5b-bb22-f281df8f7337/payload-processing/0.log" Nov 03 21:25:44.575543 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:44.575517 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj_cd915ca2-fb6f-4a5b-bb22-f281df8f7337/payload-processing/0.log" Nov 03 21:25:56.348616 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:56.348577 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr"] Nov 03 21:25:56.349209 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:56.349192 5615 cpu_manager.go:414] "containerMap: removing container" podUID="a41dcb03-3288-4425-866e-4473c432b8b5" containerName="istio-proxy" Nov 03 21:25:56.349263 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:56.349210 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a41dcb03-3288-4425-866e-4473c432b8b5" containerName="istio-proxy" Nov 03 21:25:56.349263 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:56.349235 5615 cpu_manager.go:414] "containerMap: removing container" podUID="314a2820-1a0c-477b-adab-a2428c552b65" containerName="maas-api" Nov 03 21:25:56.349263 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:56.349240 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="314a2820-1a0c-477b-adab-a2428c552b65" containerName="maas-api" Nov 03 21:25:56.349378 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:56.349344 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="314a2820-1a0c-477b-adab-a2428c552b65" containerName="maas-api" Nov 03 21:25:56.349378 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:56.349355 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="a41dcb03-3288-4425-866e-4473c432b8b5" containerName="istio-proxy" Nov 03 21:25:56.349378 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:56.349366 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="a41dcb03-3288-4425-866e-4473c432b8b5" containerName="istio-proxy" Nov 03 21:25:56.352929 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:56.352908 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr" Nov 03 21:25:56.452474 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:56.452436 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-fe2df9-serving-cert\"" Nov 03 21:25:56.454961 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:56.454932 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr"] Nov 03 21:25:56.511241 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:56.511197 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftnqz\" (UniqueName: \"kubernetes.io/projected/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-kube-api-access-ftnqz\") pod \"maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr\" (UID: \"8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr" Nov 03 21:25:56.511419 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:56.511265 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr\" (UID: \"8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr" Nov 03 21:25:56.511465 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:56.511410 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-metrics-certs\") pod \"maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr\" (UID: \"8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr" Nov 03 21:25:56.511594 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:56.511570 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr\" (UID: \"8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr" Nov 03 21:25:56.614236 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:56.613900 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftnqz\" (UniqueName: \"kubernetes.io/projected/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-kube-api-access-ftnqz\") pod \"maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr\" (UID: \"8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr" Nov 03 21:25:56.614236 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:56.613994 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr\" (UID: \"8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr" Nov 03 21:25:56.614236 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:56.614068 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-metrics-certs\") pod \"maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr\" (UID: \"8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr" Nov 03 21:25:56.614236 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:56.614211 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr\" (UID: \"8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr" Nov 03 21:25:56.614818 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:25:56.614393 5615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-serving-cert: secret "maas-api-e2e-praxis-b-fe2df9-serving-cert" not found Nov 03 21:25:56.614818 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:25:56.614460 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-maas-api-tls podName:8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a nodeName:}" failed. No retries permitted until 2026-11-03 21:25:57.11443743 +0000 UTC m=+2251.582216156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-maas-api-tls") pod "maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr" (UID: "8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a") : secret "maas-api-e2e-praxis-b-fe2df9-serving-cert" not found Nov 03 21:25:56.618922 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:56.615697 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr\" (UID: \"8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr" Nov 03 21:25:56.621942 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:56.621911 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-metrics-certs\") pod \"maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr\" (UID: \"8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr" Nov 03 21:25:56.673269 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:56.673236 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftnqz\" (UniqueName: \"kubernetes.io/projected/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-kube-api-access-ftnqz\") pod \"maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr\" (UID: \"8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr" Nov 03 21:25:57.121153 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:57.121106 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr\" (UID: \"8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr" Nov 03 21:25:57.123583 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:57.123553 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr\" (UID: \"8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr" Nov 03 21:25:57.267087 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:57.267054 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr" Nov 03 21:25:57.638679 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:57.638641 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr"] Nov 03 21:25:57.640523 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:25:57.640488 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cfa7b2e_e2b8_435e_94dc_37fb2fdd469a.slice/crio-c903c341a4947063c69db283f3b8fd7770b1fdb945ce73cc4c7295ccf1ed4a79 WatchSource:0}: Error finding container c903c341a4947063c69db283f3b8fd7770b1fdb945ce73cc4c7295ccf1ed4a79: Status 404 returned error can't find the container with id c903c341a4947063c69db283f3b8fd7770b1fdb945ce73cc4c7295ccf1ed4a79 Nov 03 21:25:57.641936 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:57.641915 5615 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:25:58.535938 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:58.535898 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr" event={"ID":"8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a","Type":"ContainerStarted","Data":"c903c341a4947063c69db283f3b8fd7770b1fdb945ce73cc4c7295ccf1ed4a79"} Nov 03 21:25:59.544232 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:59.544188 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr" event={"ID":"8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a","Type":"ContainerStarted","Data":"87f4374e9ebbb843edcca4180cea9aa7c60f3ba49baebc28dddf31f5c6f1d494"} Nov 03 21:25:59.544644 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:59.544404 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr" Nov 03 21:25:59.657972 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:25:59.657917 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr" podStartSLOduration=2.669150883 podStartE2EDuration="3.657903799s" totalImagesPullingTime="988.752916ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:59.656413051 +0000 UTC m=+2254.124191775" watchObservedRunningTime="2026-11-03 21:25:59.657903799 +0000 UTC m=+2254.125682522" Nov 03 21:26:05.556185 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:05.556105 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr" Nov 03 21:26:10.621515 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:26:10.621468 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:26:16.445816 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:16.445781 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6"] Nov 03 21:26:16.446424 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:16.446001 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" podUID="d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1" containerName="istio-proxy" containerID="cri-o://d50cce2819f0cc2a8116d79740ccd9cf8b8734cd5fc015afa8ece7d9883683ee" gracePeriod=30 Nov 03 21:26:21.816526 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.816491 5615 generic.go:356] "Generic (PLEG): container finished" podID="d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1" containerID="d50cce2819f0cc2a8116d79740ccd9cf8b8734cd5fc015afa8ece7d9883683ee" exitCode=0 Nov 03 21:26:21.816903 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.816565 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" event={"ID":"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1","Type":"ContainerDied","Data":"d50cce2819f0cc2a8116d79740ccd9cf8b8734cd5fc015afa8ece7d9883683ee"} Nov 03 21:26:21.816903 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.816686 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" event={"ID":"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1","Type":"ContainerDied","Data":"f2e64e91ca19a27af0bb0a35599e9930a2c9ef42e3df7b1fc4598812d1fdab44"} Nov 03 21:26:21.816903 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.816717 5615 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2e64e91ca19a27af0bb0a35599e9930a2c9ef42e3df7b1fc4598812d1fdab44" Nov 03 21:26:21.816903 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.816746 5615 scope.go:122] "RemoveContainer" containerID="e64f9e2af5fb776cb21f76dde321e501ed0e0dc217d103ad9bb140da94cf4de9" Nov 03 21:26:21.822256 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.822231 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:26:21.876017 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.875983 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-workload-socket\") pod \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " Nov 03 21:26:21.876017 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.876022 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istio-envoy\") pod \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " Nov 03 21:26:21.876274 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.876118 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istio-token\" (UniqueName: \"kubernetes.io/projected/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istio-token\") pod \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " Nov 03 21:26:21.876274 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.876148 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istio-data\") pod \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " Nov 03 21:26:21.876274 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.876190 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-kube-api-access-vzrrk\" (UniqueName: \"kubernetes.io/projected/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-kube-api-access-vzrrk\") pod \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " Nov 03 21:26:21.876274 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.876231 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-credential-socket\") pod \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " Nov 03 21:26:21.876496 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.876292 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istio-podinfo\") pod \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " Nov 03 21:26:21.876496 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.876323 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-workload-certs\") pod \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " Nov 03 21:26:21.876496 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.876354 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istiod-ca-cert\") pod \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\" (UID: \"d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1\") " Nov 03 21:26:21.876496 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.876359 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-workload-socket" pod "d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1" (UID: "d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:21.876496 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.876481 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istio-data" pod "d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1" (UID: "d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:21.876822 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.876573 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-workload-certs" pod "d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1" (UID: "d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:21.876921 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.876842 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istiod-ca-cert" pod "d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1" (UID: "d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:26:21.876921 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.876914 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-credential-socket" pod "d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1" (UID: "d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:21.877089 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.877039 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:21.877089 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.877064 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:21.877215 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.877102 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:21.877215 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.877118 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:21.877215 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.877132 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:21.878497 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.878467 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istio-envoy" pod "d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1" (UID: "d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:21.878624 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.878606 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istio-token" pod "d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1" (UID: "d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:26:21.878624 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.878612 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istio-podinfo" pod "d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1" (UID: "d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:26:21.878710 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.878694 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-kube-api-access-vzrrk" pod "d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1" (UID: "d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1"). InnerVolumeSpecName "kube-api-access-vzrrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:26:21.977887 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.977846 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:21.977887 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.977885 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:21.978111 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.977899 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:21.978111 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:21.977912 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzrrk\" (UniqueName: \"kubernetes.io/projected/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1-kube-api-access-vzrrk\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:22.844101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:22.838450 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6" Nov 03 21:26:23.160260 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:23.160171 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6"] Nov 03 21:26:23.160881 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:23.160862 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-1190cc-openshift-default-68f568dbbf-wmzb6"] Nov 03 21:26:23.652170 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:23.652135 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1" path="/var/lib/kubelet/pods/d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1/volumes" Nov 03 21:26:32.969287 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:32.969251 5615 generic.go:356] "Generic (PLEG): container finished" podID="69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87" containerID="37c59b60e2c16777df037693df6ddad73ffb754cd04ed153ec359cd37fcf4ee0" exitCode=0 Nov 03 21:26:32.969847 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:32.969307 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" event={"ID":"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87","Type":"ContainerDied","Data":"37c59b60e2c16777df037693df6ddad73ffb754cd04ed153ec359cd37fcf4ee0"} Nov 03 21:26:32.969924 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:32.969842 5615 scope.go:122] "RemoveContainer" containerID="37c59b60e2c16777df037693df6ddad73ffb754cd04ed153ec359cd37fcf4ee0" Nov 03 21:26:32.971655 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:32.971618 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:26:32.971757 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:32.971701 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:26:32.971757 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:32.971734 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:26:33.977771 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:33.977734 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" event={"ID":"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87","Type":"ContainerStarted","Data":"e4096b6622a61dff89ca87353aef901d1b3f5de38edb3e69bd2eb56fab72a87b"} Nov 03 21:26:34.953126 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:34.953062 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:26:34.954452 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:34.954429 5615 patch_prober.go:29] interesting pod/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 21:26:34.954531 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:34.954479 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" podUID="69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 21:26:35.953204 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:35.953172 5615 patch_prober.go:29] interesting pod/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 21:26:35.953627 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:35.953223 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" podUID="69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 21:26:36.953401 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:36.953358 5615 patch_prober.go:29] interesting pod/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 21:26:36.953908 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:36.953421 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" podUID="69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 21:26:36.976474 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:36.976440 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:26:37.952707 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:37.952671 5615 patch_prober.go:29] interesting pod/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 21:26:37.952897 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:37.952721 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" podUID="69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 21:26:38.953634 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:38.953587 5615 patch_prober.go:29] interesting pod/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 21:26:38.954257 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:38.953649 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" podUID="69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 21:26:39.958209 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:39.958173 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:26:39.959529 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:39.959499 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:26:40.619282 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:26:40.619240 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:26:54.081159 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:54.081119 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj"] Nov 03 21:26:54.081758 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:54.081354 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" podUID="b3a64ee6-7833-4ab0-b8cb-ee371e0a972e" containerName="istio-proxy" containerID="cri-o://58d6569a234383c78b2774939e2adeffa9788e09c8275ae7d0eb23fee7ed668b" gracePeriod=30 Nov 03 21:26:54.128295 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:54.128244 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g"] Nov 03 21:26:54.128497 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:54.128475 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" podUID="b387372c-edf1-4d12-8460-18d61fa396ba" containerName="istio-proxy" containerID="cri-o://e904b40f35defab7aff6a9eb816c32ab891b055d1bd217ada58c5b15b179ba42" gracePeriod=30 Nov 03 21:26:54.130607 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:54.130579 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2"] Nov 03 21:26:54.130762 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:54.130740 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" podUID="889d1dcf-c570-4151-9884-ddc9962a4c0e" containerName="istio-proxy" containerID="cri-o://c91eec024676eb3ba61073787252ef5cb26556f7d082bc7f891d93a4da9d2c81" gracePeriod=30 Nov 03 21:26:54.177211 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:54.177174 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg"] Nov 03 21:26:54.177450 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:54.177423 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" podUID="124fd847-9e0a-45d1-ae6f-d3fdc6f62433" containerName="istio-proxy" containerID="cri-o://2ebafe5c26ba1c80310ecd328dc0700d39f1b83795051faa69c98004e818cc3a" gracePeriod=30 Nov 03 21:26:54.342450 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:54.342364 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:26:54.342450 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:54.342434 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" podUID="889d1dcf-c570-4151-9884-ddc9962a4c0e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:26:59.428461 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.428427 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:26:59.470503 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.470467 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-kube-api-access-9sswv\" (UniqueName: \"kubernetes.io/projected/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-kube-api-access-9sswv\") pod \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " Nov 03 21:26:59.470692 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.470538 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istiod-ca-cert\") pod \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " Nov 03 21:26:59.470692 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.470601 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istio-podinfo\") pod \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " Nov 03 21:26:59.470692 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.470646 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-workload-certs\") pod \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " Nov 03 21:26:59.470692 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.470676 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-workload-socket\") pod \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " Nov 03 21:26:59.470877 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.470748 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-credential-socket\") pod \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " Nov 03 21:26:59.470877 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.470790 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istio-data\") pod \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " Nov 03 21:26:59.470877 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.470846 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istio-envoy\") pod \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " Nov 03 21:26:59.471035 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.470901 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istio-token\" (UniqueName: \"kubernetes.io/projected/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istio-token\") pod \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\" (UID: \"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e\") " Nov 03 21:26:59.472768 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.471636 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-credential-socket" pod "b3a64ee6-7833-4ab0-b8cb-ee371e0a972e" (UID: "b3a64ee6-7833-4ab0-b8cb-ee371e0a972e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:59.472768 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.471695 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-workload-certs" pod "b3a64ee6-7833-4ab0-b8cb-ee371e0a972e" (UID: "b3a64ee6-7833-4ab0-b8cb-ee371e0a972e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:59.472768 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.472196 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.472768 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.472219 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.472768 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.472397 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-workload-socket" pod "b3a64ee6-7833-4ab0-b8cb-ee371e0a972e" (UID: "b3a64ee6-7833-4ab0-b8cb-ee371e0a972e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:59.472768 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.472472 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istio-data" pod "b3a64ee6-7833-4ab0-b8cb-ee371e0a972e" (UID: "b3a64ee6-7833-4ab0-b8cb-ee371e0a972e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:59.472768 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.472613 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istiod-ca-cert" pod "b3a64ee6-7833-4ab0-b8cb-ee371e0a972e" (UID: "b3a64ee6-7833-4ab0-b8cb-ee371e0a972e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:26:59.473247 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.472942 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:26:59.473367 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.473318 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-kube-api-access-9sswv" pod "b3a64ee6-7833-4ab0-b8cb-ee371e0a972e" (UID: "b3a64ee6-7833-4ab0-b8cb-ee371e0a972e"). InnerVolumeSpecName "kube-api-access-9sswv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:26:59.478118 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.474974 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istio-podinfo" pod "b3a64ee6-7833-4ab0-b8cb-ee371e0a972e" (UID: "b3a64ee6-7833-4ab0-b8cb-ee371e0a972e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:26:59.478118 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.475054 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istio-envoy" pod "b3a64ee6-7833-4ab0-b8cb-ee371e0a972e" (UID: "b3a64ee6-7833-4ab0-b8cb-ee371e0a972e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:59.478118 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.475947 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istio-token" pod "b3a64ee6-7833-4ab0-b8cb-ee371e0a972e" (UID: "b3a64ee6-7833-4ab0-b8cb-ee371e0a972e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:26:59.529039 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.529006 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 21:26:59.534725 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.534696 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:26:59.572897 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.572861 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-istio-envoy\") pod \"b387372c-edf1-4d12-8460-18d61fa396ba\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " Nov 03 21:26:59.573062 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.572908 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istio-data\") pod \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " Nov 03 21:26:59.573062 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.572932 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-workload-socket\") pod \"889d1dcf-c570-4151-9884-ddc9962a4c0e\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " Nov 03 21:26:59.573062 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.572953 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-workload-certs\") pod \"889d1dcf-c570-4151-9884-ddc9962a4c0e\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " Nov 03 21:26:59.573062 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.572986 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istio-podinfo\") pod \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " Nov 03 21:26:59.573062 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.573046 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-credential-socket\") pod \"889d1dcf-c570-4151-9884-ddc9962a4c0e\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " Nov 03 21:26:59.573384 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.573116 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b387372c-edf1-4d12-8460-18d61fa396ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b387372c-edf1-4d12-8460-18d61fa396ba-istiod-ca-cert\") pod \"b387372c-edf1-4d12-8460-18d61fa396ba\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " Nov 03 21:26:59.573384 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.573161 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-credential-socket\") pod \"b387372c-edf1-4d12-8460-18d61fa396ba\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " Nov 03 21:26:59.573384 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.573206 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/889d1dcf-c570-4151-9884-ddc9962a4c0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/889d1dcf-c570-4151-9884-ddc9962a4c0e-istio-podinfo\") pod \"889d1dcf-c570-4151-9884-ddc9962a4c0e\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " Nov 03 21:26:59.573384 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.573233 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/889d1dcf-c570-4151-9884-ddc9962a4c0e-istio-token\" (UniqueName: \"kubernetes.io/projected/889d1dcf-c570-4151-9884-ddc9962a4c0e-istio-token\") pod \"889d1dcf-c570-4151-9884-ddc9962a4c0e\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " Nov 03 21:26:59.573384 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.573273 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-credential-socket\") pod \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " Nov 03 21:26:59.573384 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.573285 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-workload-socket" pod "889d1dcf-c570-4151-9884-ddc9962a4c0e" (UID: "889d1dcf-c570-4151-9884-ddc9962a4c0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:59.573384 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.573302 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-workload-certs" pod "889d1dcf-c570-4151-9884-ddc9962a4c0e" (UID: "889d1dcf-c570-4151-9884-ddc9962a4c0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:59.573384 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.573348 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istio-envoy\") pod \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " Nov 03 21:26:59.573384 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.573375 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/889d1dcf-c570-4151-9884-ddc9962a4c0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/889d1dcf-c570-4151-9884-ddc9962a4c0e-istiod-ca-cert\") pod \"889d1dcf-c570-4151-9884-ddc9962a4c0e\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " Nov 03 21:26:59.573863 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.573405 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istiod-ca-cert\") pod \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " Nov 03 21:26:59.573863 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.573438 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b387372c-edf1-4d12-8460-18d61fa396ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b387372c-edf1-4d12-8460-18d61fa396ba-istio-podinfo\") pod \"b387372c-edf1-4d12-8460-18d61fa396ba\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " Nov 03 21:26:59.573863 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.573478 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-workload-certs\") pod \"b387372c-edf1-4d12-8460-18d61fa396ba\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " Nov 03 21:26:59.573863 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.573520 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-credential-socket" pod "889d1dcf-c570-4151-9884-ddc9962a4c0e" (UID: "889d1dcf-c570-4151-9884-ddc9962a4c0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:59.573863 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.573547 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-istio-envoy\") pod \"889d1dcf-c570-4151-9884-ddc9962a4c0e\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " Nov 03 21:26:59.573863 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.573566 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-credential-socket" pod "b387372c-edf1-4d12-8460-18d61fa396ba" (UID: "b387372c-edf1-4d12-8460-18d61fa396ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:59.573863 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.573590 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-workload-socket\") pod \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " Nov 03 21:26:59.573863 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.573609 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b387372c-edf1-4d12-8460-18d61fa396ba-istiod-ca-cert" pod "b387372c-edf1-4d12-8460-18d61fa396ba" (UID: "b387372c-edf1-4d12-8460-18d61fa396ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:26:59.573863 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.573628 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-kube-api-access-dckxc\" (UniqueName: \"kubernetes.io/projected/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-kube-api-access-dckxc\") pod \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " Nov 03 21:26:59.573863 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.573689 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istio-token\" (UniqueName: \"kubernetes.io/projected/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istio-token\") pod \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " Nov 03 21:26:59.573863 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.573740 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-workload-certs\") pod \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\" (UID: \"124fd847-9e0a-45d1-ae6f-d3fdc6f62433\") " Nov 03 21:26:59.573863 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.573775 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-istio-data\") pod \"b387372c-edf1-4d12-8460-18d61fa396ba\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " Nov 03 21:26:59.573863 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.573802 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-istio-data\") pod \"889d1dcf-c570-4151-9884-ddc9962a4c0e\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " Nov 03 21:26:59.573863 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.573840 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b387372c-edf1-4d12-8460-18d61fa396ba-kube-api-access-d5cf5\" (UniqueName: \"kubernetes.io/projected/b387372c-edf1-4d12-8460-18d61fa396ba-kube-api-access-d5cf5\") pod \"b387372c-edf1-4d12-8460-18d61fa396ba\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " Nov 03 21:26:59.574740 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.573874 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-workload-socket\") pod \"b387372c-edf1-4d12-8460-18d61fa396ba\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " Nov 03 21:26:59.574740 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.573891 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-credential-socket" pod "124fd847-9e0a-45d1-ae6f-d3fdc6f62433" (UID: "124fd847-9e0a-45d1-ae6f-d3fdc6f62433"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:59.574740 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.573909 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/889d1dcf-c570-4151-9884-ddc9962a4c0e-kube-api-access-f4qzg\" (UniqueName: \"kubernetes.io/projected/889d1dcf-c570-4151-9884-ddc9962a4c0e-kube-api-access-f4qzg\") pod \"889d1dcf-c570-4151-9884-ddc9962a4c0e\" (UID: \"889d1dcf-c570-4151-9884-ddc9962a4c0e\") " Nov 03 21:26:59.574740 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.574224 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istio-data" pod "124fd847-9e0a-45d1-ae6f-d3fdc6f62433" (UID: "124fd847-9e0a-45d1-ae6f-d3fdc6f62433"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:59.575917 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.575884 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istio-podinfo" pod "124fd847-9e0a-45d1-ae6f-d3fdc6f62433" (UID: "124fd847-9e0a-45d1-ae6f-d3fdc6f62433"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:26:59.576278 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.576252 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/889d1dcf-c570-4151-9884-ddc9962a4c0e-istiod-ca-cert" pod "889d1dcf-c570-4151-9884-ddc9962a4c0e" (UID: "889d1dcf-c570-4151-9884-ddc9962a4c0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:26:59.576382 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.576287 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istio-envoy" pod "124fd847-9e0a-45d1-ae6f-d3fdc6f62433" (UID: "124fd847-9e0a-45d1-ae6f-d3fdc6f62433"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:59.576463 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.576435 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/889d1dcf-c570-4151-9884-ddc9962a4c0e-kube-api-access-f4qzg" pod "889d1dcf-c570-4151-9884-ddc9962a4c0e" (UID: "889d1dcf-c570-4151-9884-ddc9962a4c0e"). InnerVolumeSpecName "kube-api-access-f4qzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:26:59.576527 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.576500 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/889d1dcf-c570-4151-9884-ddc9962a4c0e-istio-podinfo" pod "889d1dcf-c570-4151-9884-ddc9962a4c0e" (UID: "889d1dcf-c570-4151-9884-ddc9962a4c0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:26:59.576621 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.576597 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istiod-ca-cert" pod "124fd847-9e0a-45d1-ae6f-d3fdc6f62433" (UID: "124fd847-9e0a-45d1-ae6f-d3fdc6f62433"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:26:59.576741 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.576720 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-istio-data" pod "889d1dcf-c570-4151-9884-ddc9962a4c0e" (UID: "889d1dcf-c570-4151-9884-ddc9962a4c0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:59.576864 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.576840 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-istio-envoy" pod "b387372c-edf1-4d12-8460-18d61fa396ba" (UID: "b387372c-edf1-4d12-8460-18d61fa396ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:59.577122 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.576928 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-istio-data" pod "b387372c-edf1-4d12-8460-18d61fa396ba" (UID: "b387372c-edf1-4d12-8460-18d61fa396ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:59.577122 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.576967 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/889d1dcf-c570-4151-9884-ddc9962a4c0e-istio-token" pod "889d1dcf-c570-4151-9884-ddc9962a4c0e" (UID: "889d1dcf-c570-4151-9884-ddc9962a4c0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:26:59.577261 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.577123 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-kube-api-access-dckxc" pod "124fd847-9e0a-45d1-ae6f-d3fdc6f62433" (UID: "124fd847-9e0a-45d1-ae6f-d3fdc6f62433"). InnerVolumeSpecName "kube-api-access-dckxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:26:59.577261 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.577135 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-workload-certs" pod "124fd847-9e0a-45d1-ae6f-d3fdc6f62433" (UID: "124fd847-9e0a-45d1-ae6f-d3fdc6f62433"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:59.577261 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.577178 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b387372c-edf1-4d12-8460-18d61fa396ba-istio-token\" (UniqueName: \"kubernetes.io/projected/b387372c-edf1-4d12-8460-18d61fa396ba-istio-token\") pod \"b387372c-edf1-4d12-8460-18d61fa396ba\" (UID: \"b387372c-edf1-4d12-8460-18d61fa396ba\") " Nov 03 21:26:59.577410 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.577390 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-workload-socket" pod "b387372c-edf1-4d12-8460-18d61fa396ba" (UID: "b387372c-edf1-4d12-8460-18d61fa396ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:59.577464 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.577407 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-workload-certs" pod "b387372c-edf1-4d12-8460-18d61fa396ba" (UID: "b387372c-edf1-4d12-8460-18d61fa396ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:59.577849 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.577825 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.577953 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.577852 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/889d1dcf-c570-4151-9884-ddc9962a4c0e-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.577953 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.577869 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/889d1dcf-c570-4151-9884-ddc9962a4c0e-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.577953 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.577885 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.577953 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.577900 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.577953 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.577914 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/889d1dcf-c570-4151-9884-ddc9962a4c0e-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.577953 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.577930 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.577953 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.577945 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.578482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.577959 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.578482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.577974 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.578482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.577988 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.578482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.578001 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dckxc\" (UniqueName: \"kubernetes.io/projected/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-kube-api-access-dckxc\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.578482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.578016 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.578482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.578030 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.578482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.578044 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.578482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.578057 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.578482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.578070 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.578482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.578101 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.578482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.578115 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4qzg\" (UniqueName: \"kubernetes.io/projected/889d1dcf-c570-4151-9884-ddc9962a4c0e-kube-api-access-f4qzg\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.578482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.578129 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.578482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.578144 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.578482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.578157 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.578482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.578172 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.578482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.578187 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.578482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.578202 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.578482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.578216 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b387372c-edf1-4d12-8460-18d61fa396ba-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.578482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.578231 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b387372c-edf1-4d12-8460-18d61fa396ba-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.578482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.578247 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sswv\" (UniqueName: \"kubernetes.io/projected/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e-kube-api-access-9sswv\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.578482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.578316 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-workload-socket" pod "124fd847-9e0a-45d1-ae6f-d3fdc6f62433" (UID: "124fd847-9e0a-45d1-ae6f-d3fdc6f62433"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:59.579251 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.578536 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b387372c-edf1-4d12-8460-18d61fa396ba-istio-podinfo" pod "b387372c-edf1-4d12-8460-18d61fa396ba" (UID: "b387372c-edf1-4d12-8460-18d61fa396ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:26:59.579251 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.578826 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-istio-envoy" pod "889d1dcf-c570-4151-9884-ddc9962a4c0e" (UID: "889d1dcf-c570-4151-9884-ddc9962a4c0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:59.579251 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.578859 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istio-token" pod "124fd847-9e0a-45d1-ae6f-d3fdc6f62433" (UID: "124fd847-9e0a-45d1-ae6f-d3fdc6f62433"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:26:59.579251 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.579067 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b387372c-edf1-4d12-8460-18d61fa396ba-kube-api-access-d5cf5" pod "b387372c-edf1-4d12-8460-18d61fa396ba" (UID: "b387372c-edf1-4d12-8460-18d61fa396ba"). InnerVolumeSpecName "kube-api-access-d5cf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:26:59.579375 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.579289 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b387372c-edf1-4d12-8460-18d61fa396ba-istio-token" pod "b387372c-edf1-4d12-8460-18d61fa396ba" (UID: "b387372c-edf1-4d12-8460-18d61fa396ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:26:59.679273 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.679237 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5cf5\" (UniqueName: \"kubernetes.io/projected/b387372c-edf1-4d12-8460-18d61fa396ba-kube-api-access-d5cf5\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.679273 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.679265 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b387372c-edf1-4d12-8460-18d61fa396ba-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.679273 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.679275 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b387372c-edf1-4d12-8460-18d61fa396ba-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.679521 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.679284 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/889d1dcf-c570-4151-9884-ddc9962a4c0e-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.679521 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.679306 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:59.679521 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:26:59.679325 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/124fd847-9e0a-45d1-ae6f-d3fdc6f62433-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:00.167448 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.167407 5615 generic.go:356] "Generic (PLEG): container finished" podID="889d1dcf-c570-4151-9884-ddc9962a4c0e" containerID="c91eec024676eb3ba61073787252ef5cb26556f7d082bc7f891d93a4da9d2c81" exitCode=0 Nov 03 21:27:00.167653 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.167462 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" Nov 03 21:27:00.167653 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.167499 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" event={"ID":"889d1dcf-c570-4151-9884-ddc9962a4c0e","Type":"ContainerDied","Data":"c91eec024676eb3ba61073787252ef5cb26556f7d082bc7f891d93a4da9d2c81"} Nov 03 21:27:00.167653 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.167557 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2" event={"ID":"889d1dcf-c570-4151-9884-ddc9962a4c0e","Type":"ContainerDied","Data":"4e1606c9668002adb0fc480cc42e50e38c7c3d56b6dc1f74ed07d050cba1b9ef"} Nov 03 21:27:00.167653 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.167586 5615 scope.go:122] "RemoveContainer" containerID="c91eec024676eb3ba61073787252ef5cb26556f7d082bc7f891d93a4da9d2c81" Nov 03 21:27:00.168968 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.168945 5615 generic.go:356] "Generic (PLEG): container finished" podID="124fd847-9e0a-45d1-ae6f-d3fdc6f62433" containerID="2ebafe5c26ba1c80310ecd328dc0700d39f1b83795051faa69c98004e818cc3a" exitCode=0 Nov 03 21:27:00.169089 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.168995 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" Nov 03 21:27:00.169089 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.169036 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" event={"ID":"124fd847-9e0a-45d1-ae6f-d3fdc6f62433","Type":"ContainerDied","Data":"2ebafe5c26ba1c80310ecd328dc0700d39f1b83795051faa69c98004e818cc3a"} Nov 03 21:27:00.169211 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.169113 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg" event={"ID":"124fd847-9e0a-45d1-ae6f-d3fdc6f62433","Type":"ContainerDied","Data":"3382e6815bdcb5c2a4fc68b97a0810af6cb4282062e21a2d51101940934d7785"} Nov 03 21:27:00.170736 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.170702 5615 generic.go:356] "Generic (PLEG): container finished" podID="b3a64ee6-7833-4ab0-b8cb-ee371e0a972e" containerID="58d6569a234383c78b2774939e2adeffa9788e09c8275ae7d0eb23fee7ed668b" exitCode=0 Nov 03 21:27:00.170842 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.170755 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" Nov 03 21:27:00.170842 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.170754 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" event={"ID":"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e","Type":"ContainerDied","Data":"58d6569a234383c78b2774939e2adeffa9788e09c8275ae7d0eb23fee7ed668b"} Nov 03 21:27:00.170842 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.170805 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj" event={"ID":"b3a64ee6-7833-4ab0-b8cb-ee371e0a972e","Type":"ContainerDied","Data":"a73ae5943c14d19d46185e3f90c98829a3b60f71534f8070b28fcaa14cfde2e8"} Nov 03 21:27:00.172411 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.172389 5615 generic.go:356] "Generic (PLEG): container finished" podID="b387372c-edf1-4d12-8460-18d61fa396ba" containerID="e904b40f35defab7aff6a9eb816c32ab891b055d1bd217ada58c5b15b179ba42" exitCode=0 Nov 03 21:27:00.172516 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.172429 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" Nov 03 21:27:00.172516 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.172448 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" event={"ID":"b387372c-edf1-4d12-8460-18d61fa396ba","Type":"ContainerDied","Data":"e904b40f35defab7aff6a9eb816c32ab891b055d1bd217ada58c5b15b179ba42"} Nov 03 21:27:00.172516 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.172482 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g" event={"ID":"b387372c-edf1-4d12-8460-18d61fa396ba","Type":"ContainerDied","Data":"63ff54653131f55a3e9f7613255af7c83315233e99a4ac21cf72fb941666e057"} Nov 03 21:27:00.179328 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.179308 5615 scope.go:122] "RemoveContainer" containerID="c91eec024676eb3ba61073787252ef5cb26556f7d082bc7f891d93a4da9d2c81" Nov 03 21:27:00.179598 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:27:00.179578 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c91eec024676eb3ba61073787252ef5cb26556f7d082bc7f891d93a4da9d2c81\": container with ID starting with c91eec024676eb3ba61073787252ef5cb26556f7d082bc7f891d93a4da9d2c81 not found: ID does not exist" containerID="c91eec024676eb3ba61073787252ef5cb26556f7d082bc7f891d93a4da9d2c81" Nov 03 21:27:00.179644 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.179605 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c91eec024676eb3ba61073787252ef5cb26556f7d082bc7f891d93a4da9d2c81"} err="failed to get container status \"c91eec024676eb3ba61073787252ef5cb26556f7d082bc7f891d93a4da9d2c81\": rpc error: code = NotFound desc = could not find container \"c91eec024676eb3ba61073787252ef5cb26556f7d082bc7f891d93a4da9d2c81\": container with ID starting with c91eec024676eb3ba61073787252ef5cb26556f7d082bc7f891d93a4da9d2c81 not found: ID does not exist" Nov 03 21:27:00.179644 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.179622 5615 scope.go:122] "RemoveContainer" containerID="2ebafe5c26ba1c80310ecd328dc0700d39f1b83795051faa69c98004e818cc3a" Nov 03 21:27:00.188768 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.188746 5615 scope.go:122] "RemoveContainer" containerID="2ebafe5c26ba1c80310ecd328dc0700d39f1b83795051faa69c98004e818cc3a" Nov 03 21:27:00.189114 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:27:00.189043 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ebafe5c26ba1c80310ecd328dc0700d39f1b83795051faa69c98004e818cc3a\": container with ID starting with 2ebafe5c26ba1c80310ecd328dc0700d39f1b83795051faa69c98004e818cc3a not found: ID does not exist" containerID="2ebafe5c26ba1c80310ecd328dc0700d39f1b83795051faa69c98004e818cc3a" Nov 03 21:27:00.189177 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.189123 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ebafe5c26ba1c80310ecd328dc0700d39f1b83795051faa69c98004e818cc3a"} err="failed to get container status \"2ebafe5c26ba1c80310ecd328dc0700d39f1b83795051faa69c98004e818cc3a\": rpc error: code = NotFound desc = could not find container \"2ebafe5c26ba1c80310ecd328dc0700d39f1b83795051faa69c98004e818cc3a\": container with ID starting with 2ebafe5c26ba1c80310ecd328dc0700d39f1b83795051faa69c98004e818cc3a not found: ID does not exist" Nov 03 21:27:00.189177 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.189152 5615 scope.go:122] "RemoveContainer" containerID="58d6569a234383c78b2774939e2adeffa9788e09c8275ae7d0eb23fee7ed668b" Nov 03 21:27:00.198445 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.198425 5615 scope.go:122] "RemoveContainer" containerID="58d6569a234383c78b2774939e2adeffa9788e09c8275ae7d0eb23fee7ed668b" Nov 03 21:27:00.198726 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:27:00.198706 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58d6569a234383c78b2774939e2adeffa9788e09c8275ae7d0eb23fee7ed668b\": container with ID starting with 58d6569a234383c78b2774939e2adeffa9788e09c8275ae7d0eb23fee7ed668b not found: ID does not exist" containerID="58d6569a234383c78b2774939e2adeffa9788e09c8275ae7d0eb23fee7ed668b" Nov 03 21:27:00.198800 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.198737 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58d6569a234383c78b2774939e2adeffa9788e09c8275ae7d0eb23fee7ed668b"} err="failed to get container status \"58d6569a234383c78b2774939e2adeffa9788e09c8275ae7d0eb23fee7ed668b\": rpc error: code = NotFound desc = could not find container \"58d6569a234383c78b2774939e2adeffa9788e09c8275ae7d0eb23fee7ed668b\": container with ID starting with 58d6569a234383c78b2774939e2adeffa9788e09c8275ae7d0eb23fee7ed668b not found: ID does not exist" Nov 03 21:27:00.198800 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.198761 5615 scope.go:122] "RemoveContainer" containerID="e904b40f35defab7aff6a9eb816c32ab891b055d1bd217ada58c5b15b179ba42" Nov 03 21:27:00.207806 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.207787 5615 scope.go:122] "RemoveContainer" containerID="e904b40f35defab7aff6a9eb816c32ab891b055d1bd217ada58c5b15b179ba42" Nov 03 21:27:00.208045 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:27:00.208029 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e904b40f35defab7aff6a9eb816c32ab891b055d1bd217ada58c5b15b179ba42\": container with ID starting with e904b40f35defab7aff6a9eb816c32ab891b055d1bd217ada58c5b15b179ba42 not found: ID does not exist" containerID="e904b40f35defab7aff6a9eb816c32ab891b055d1bd217ada58c5b15b179ba42" Nov 03 21:27:00.208111 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.208053 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e904b40f35defab7aff6a9eb816c32ab891b055d1bd217ada58c5b15b179ba42"} err="failed to get container status \"e904b40f35defab7aff6a9eb816c32ab891b055d1bd217ada58c5b15b179ba42\": rpc error: code = NotFound desc = could not find container \"e904b40f35defab7aff6a9eb816c32ab891b055d1bd217ada58c5b15b179ba42\": container with ID starting with e904b40f35defab7aff6a9eb816c32ab891b055d1bd217ada58c5b15b179ba42 not found: ID does not exist" Nov 03 21:27:00.325108 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.325053 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g"] Nov 03 21:27:00.327678 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.327653 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-6fs4g"] Nov 03 21:27:00.430943 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.430855 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg"] Nov 03 21:27:00.435021 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.434987 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-t7wzg"] Nov 03 21:27:00.539045 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.539009 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj"] Nov 03 21:27:00.543348 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.543317 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9qjbj"] Nov 03 21:27:00.647371 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.647336 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2"] Nov 03 21:27:00.649658 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:00.649626 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4zxf2"] Nov 03 21:27:01.652528 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:01.652497 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="124fd847-9e0a-45d1-ae6f-d3fdc6f62433" path="/var/lib/kubelet/pods/124fd847-9e0a-45d1-ae6f-d3fdc6f62433/volumes" Nov 03 21:27:01.652990 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:01.652976 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="889d1dcf-c570-4151-9884-ddc9962a4c0e" path="/var/lib/kubelet/pods/889d1dcf-c570-4151-9884-ddc9962a4c0e/volumes" Nov 03 21:27:01.653441 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:01.653427 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b387372c-edf1-4d12-8460-18d61fa396ba" path="/var/lib/kubelet/pods/b387372c-edf1-4d12-8460-18d61fa396ba/volumes" Nov 03 21:27:01.654208 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:01.654193 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3a64ee6-7833-4ab0-b8cb-ee371e0a972e" path="/var/lib/kubelet/pods/b3a64ee6-7833-4ab0-b8cb-ee371e0a972e/volumes" Nov 03 21:27:07.038501 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.038466 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g"] Nov 03 21:27:07.039037 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.039023 5615 cpu_manager.go:414] "containerMap: removing container" podUID="889d1dcf-c570-4151-9884-ddc9962a4c0e" containerName="istio-proxy" Nov 03 21:27:07.039127 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.039039 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="889d1dcf-c570-4151-9884-ddc9962a4c0e" containerName="istio-proxy" Nov 03 21:27:07.039127 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.039054 5615 cpu_manager.go:414] "containerMap: removing container" podUID="a41dcb03-3288-4425-866e-4473c432b8b5" containerName="istio-proxy" Nov 03 21:27:07.039127 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.039060 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a41dcb03-3288-4425-866e-4473c432b8b5" containerName="istio-proxy" Nov 03 21:27:07.039127 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.039093 5615 cpu_manager.go:414] "containerMap: removing container" podUID="b387372c-edf1-4d12-8460-18d61fa396ba" containerName="istio-proxy" Nov 03 21:27:07.039127 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.039099 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b387372c-edf1-4d12-8460-18d61fa396ba" containerName="istio-proxy" Nov 03 21:27:07.039127 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.039114 5615 cpu_manager.go:414] "containerMap: removing container" podUID="d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1" containerName="istio-proxy" Nov 03 21:27:07.039127 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.039119 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1" containerName="istio-proxy" Nov 03 21:27:07.039127 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.039126 5615 cpu_manager.go:414] "containerMap: removing container" podUID="124fd847-9e0a-45d1-ae6f-d3fdc6f62433" containerName="istio-proxy" Nov 03 21:27:07.039127 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.039131 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="124fd847-9e0a-45d1-ae6f-d3fdc6f62433" containerName="istio-proxy" Nov 03 21:27:07.039396 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.039138 5615 cpu_manager.go:414] "containerMap: removing container" podUID="b3a64ee6-7833-4ab0-b8cb-ee371e0a972e" containerName="istio-proxy" Nov 03 21:27:07.039396 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.039142 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3a64ee6-7833-4ab0-b8cb-ee371e0a972e" containerName="istio-proxy" Nov 03 21:27:07.039396 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.039154 5615 cpu_manager.go:414] "containerMap: removing container" podUID="d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1" containerName="istio-proxy" Nov 03 21:27:07.039396 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.039158 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1" containerName="istio-proxy" Nov 03 21:27:07.039396 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.039252 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="889d1dcf-c570-4151-9884-ddc9962a4c0e" containerName="istio-proxy" Nov 03 21:27:07.039396 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.039265 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="b387372c-edf1-4d12-8460-18d61fa396ba" containerName="istio-proxy" Nov 03 21:27:07.039396 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.039272 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1" containerName="istio-proxy" Nov 03 21:27:07.039396 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.039278 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="124fd847-9e0a-45d1-ae6f-d3fdc6f62433" containerName="istio-proxy" Nov 03 21:27:07.039396 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.039283 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3a64ee6-7833-4ab0-b8cb-ee371e0a972e" containerName="istio-proxy" Nov 03 21:27:07.039396 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.039289 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1760fbf-ca28-4bb3-949f-e59d6b1e9cb1" containerName="istio-proxy" Nov 03 21:27:07.044662 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.044636 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g" Nov 03 21:27:07.094320 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.094279 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-2285b9-serving-cert\"" Nov 03 21:27:07.145241 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.145202 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g"] Nov 03 21:27:07.167589 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.167551 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb55r\" (UniqueName: \"kubernetes.io/projected/a6cbf2de-2dc6-4747-9150-61932c13c60a-kube-api-access-mb55r\") pod \"maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g\" (UID: \"a6cbf2de-2dc6-4747-9150-61932c13c60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g" Nov 03 21:27:07.167761 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.167718 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6cbf2de-2dc6-4747-9150-61932c13c60a-metrics-certs\") pod \"maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g\" (UID: \"a6cbf2de-2dc6-4747-9150-61932c13c60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g" Nov 03 21:27:07.167761 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.167754 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6cbf2de-2dc6-4747-9150-61932c13c60a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g\" (UID: \"a6cbf2de-2dc6-4747-9150-61932c13c60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g" Nov 03 21:27:07.167852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.167781 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6cbf2de-2dc6-4747-9150-61932c13c60a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g\" (UID: \"a6cbf2de-2dc6-4747-9150-61932c13c60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g" Nov 03 21:27:07.268708 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.268673 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb55r\" (UniqueName: \"kubernetes.io/projected/a6cbf2de-2dc6-4747-9150-61932c13c60a-kube-api-access-mb55r\") pod \"maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g\" (UID: \"a6cbf2de-2dc6-4747-9150-61932c13c60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g" Nov 03 21:27:07.268919 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.268774 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6cbf2de-2dc6-4747-9150-61932c13c60a-metrics-certs\") pod \"maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g\" (UID: \"a6cbf2de-2dc6-4747-9150-61932c13c60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g" Nov 03 21:27:07.268919 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.268796 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6cbf2de-2dc6-4747-9150-61932c13c60a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g\" (UID: \"a6cbf2de-2dc6-4747-9150-61932c13c60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g" Nov 03 21:27:07.268919 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.268813 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6cbf2de-2dc6-4747-9150-61932c13c60a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g\" (UID: \"a6cbf2de-2dc6-4747-9150-61932c13c60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g" Nov 03 21:27:07.269122 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:27:07.269003 5615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-serving-cert: secret "maas-api-e2e-praxis-b-2285b9-serving-cert" not found Nov 03 21:27:07.269122 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:27:07.269084 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6cbf2de-2dc6-4747-9150-61932c13c60a-maas-api-tls podName:a6cbf2de-2dc6-4747-9150-61932c13c60a nodeName:}" failed. No retries permitted until 2026-11-03 21:27:07.769050543 +0000 UTC m=+2322.236829259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a6cbf2de-2dc6-4747-9150-61932c13c60a-maas-api-tls") pod "maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g" (UID: "a6cbf2de-2dc6-4747-9150-61932c13c60a") : secret "maas-api-e2e-praxis-b-2285b9-serving-cert" not found Nov 03 21:27:07.269709 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.269683 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6cbf2de-2dc6-4747-9150-61932c13c60a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g\" (UID: \"a6cbf2de-2dc6-4747-9150-61932c13c60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g" Nov 03 21:27:07.309064 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.308982 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6cbf2de-2dc6-4747-9150-61932c13c60a-metrics-certs\") pod \"maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g\" (UID: \"a6cbf2de-2dc6-4747-9150-61932c13c60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g" Nov 03 21:27:07.331946 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.331912 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb55r\" (UniqueName: \"kubernetes.io/projected/a6cbf2de-2dc6-4747-9150-61932c13c60a-kube-api-access-mb55r\") pod \"maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g\" (UID: \"a6cbf2de-2dc6-4747-9150-61932c13c60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g" Nov 03 21:27:07.775253 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.775212 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6cbf2de-2dc6-4747-9150-61932c13c60a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g\" (UID: \"a6cbf2de-2dc6-4747-9150-61932c13c60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g" Nov 03 21:27:07.777691 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.777661 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6cbf2de-2dc6-4747-9150-61932c13c60a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g\" (UID: \"a6cbf2de-2dc6-4747-9150-61932c13c60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g" Nov 03 21:27:07.959207 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:07.959161 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g" Nov 03 21:27:08.330106 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:08.330041 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g"] Nov 03 21:27:08.332841 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:27:08.332812 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6cbf2de_2dc6_4747_9150_61932c13c60a.slice/crio-e842487aafc7f13ba52f299e0acaf6e666d2671eaaa9cb204cf005b50d0e7c53 WatchSource:0}: Error finding container e842487aafc7f13ba52f299e0acaf6e666d2671eaaa9cb204cf005b50d0e7c53: Status 404 returned error can't find the container with id e842487aafc7f13ba52f299e0acaf6e666d2671eaaa9cb204cf005b50d0e7c53 Nov 03 21:27:09.235819 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:09.235789 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g" event={"ID":"a6cbf2de-2dc6-4747-9150-61932c13c60a","Type":"ContainerStarted","Data":"e842487aafc7f13ba52f299e0acaf6e666d2671eaaa9cb204cf005b50d0e7c53"} Nov 03 21:27:10.243049 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:10.243000 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g" event={"ID":"a6cbf2de-2dc6-4747-9150-61932c13c60a","Type":"ContainerStarted","Data":"e6235a34b11b2f50fa1088c313a73ce8c9b7ab5420ae1c352ccffaff6198d6bf"} Nov 03 21:27:10.243571 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:10.243187 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g" Nov 03 21:27:10.354998 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:10.354946 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g" podStartSLOduration=3.473141733 podStartE2EDuration="4.354931733s" totalImagesPullingTime="881.79ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:27:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:27:10.354337095 +0000 UTC m=+2324.822115818" watchObservedRunningTime="2026-11-03 21:27:10.354931733 +0000 UTC m=+2324.822710480" Nov 03 21:27:10.619677 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:27:10.619587 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:27:15.997394 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:15.997349 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d"] Nov 03 21:27:15.997818 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:15.997566 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d" podUID="342a2378-e673-42df-bc30-91af7ba1c534" containerName="maas-api" containerID="cri-o://29e687e739ace9ba589b2e71fe10b2953eabb7f2c63056827d5357ef263c9cb9" gracePeriod=30 Nov 03 21:27:16.265629 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:16.265543 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g" Nov 03 21:27:16.279452 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:16.279415 5615 generic.go:356] "Generic (PLEG): container finished" podID="342a2378-e673-42df-bc30-91af7ba1c534" containerID="29e687e739ace9ba589b2e71fe10b2953eabb7f2c63056827d5357ef263c9cb9" exitCode=0 Nov 03 21:27:16.279616 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:16.279499 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d" event={"ID":"342a2378-e673-42df-bc30-91af7ba1c534","Type":"ContainerDied","Data":"29e687e739ace9ba589b2e71fe10b2953eabb7f2c63056827d5357ef263c9cb9"} Nov 03 21:27:16.279616 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:16.279559 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d" event={"ID":"342a2378-e673-42df-bc30-91af7ba1c534","Type":"ContainerDied","Data":"2e3297f6a4b9acb12a2c6505818f2e9df53cde7a5eaee77d36dd75640a95863b"} Nov 03 21:27:16.279616 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:16.279583 5615 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e3297f6a4b9acb12a2c6505818f2e9df53cde7a5eaee77d36dd75640a95863b" Nov 03 21:27:16.297040 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:16.297007 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d" Nov 03 21:27:16.374758 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:16.374714 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/342a2378-e673-42df-bc30-91af7ba1c534-metrics-certs\" (UniqueName: \"kubernetes.io/secret/342a2378-e673-42df-bc30-91af7ba1c534-metrics-certs\") pod \"342a2378-e673-42df-bc30-91af7ba1c534\" (UID: \"342a2378-e673-42df-bc30-91af7ba1c534\") " Nov 03 21:27:16.374936 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:16.374772 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/342a2378-e673-42df-bc30-91af7ba1c534-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/342a2378-e673-42df-bc30-91af7ba1c534-openshift-service-ca\") pod \"342a2378-e673-42df-bc30-91af7ba1c534\" (UID: \"342a2378-e673-42df-bc30-91af7ba1c534\") " Nov 03 21:27:16.374936 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:16.374801 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/342a2378-e673-42df-bc30-91af7ba1c534-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/342a2378-e673-42df-bc30-91af7ba1c534-maas-api-tls\") pod \"342a2378-e673-42df-bc30-91af7ba1c534\" (UID: \"342a2378-e673-42df-bc30-91af7ba1c534\") " Nov 03 21:27:16.374936 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:16.374866 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/342a2378-e673-42df-bc30-91af7ba1c534-kube-api-access-v8xjl\" (UniqueName: \"kubernetes.io/projected/342a2378-e673-42df-bc30-91af7ba1c534-kube-api-access-v8xjl\") pod \"342a2378-e673-42df-bc30-91af7ba1c534\" (UID: \"342a2378-e673-42df-bc30-91af7ba1c534\") " Nov 03 21:27:16.375303 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:16.375258 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/342a2378-e673-42df-bc30-91af7ba1c534-openshift-service-ca" pod "342a2378-e673-42df-bc30-91af7ba1c534" (UID: "342a2378-e673-42df-bc30-91af7ba1c534"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:27:16.375485 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:16.375443 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/342a2378-e673-42df-bc30-91af7ba1c534-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:16.377200 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:16.377170 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/342a2378-e673-42df-bc30-91af7ba1c534-maas-api-tls" pod "342a2378-e673-42df-bc30-91af7ba1c534" (UID: "342a2378-e673-42df-bc30-91af7ba1c534"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:27:16.377286 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:16.377194 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/342a2378-e673-42df-bc30-91af7ba1c534-metrics-certs" pod "342a2378-e673-42df-bc30-91af7ba1c534" (UID: "342a2378-e673-42df-bc30-91af7ba1c534"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:27:16.377286 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:16.377215 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/342a2378-e673-42df-bc30-91af7ba1c534-kube-api-access-v8xjl" pod "342a2378-e673-42df-bc30-91af7ba1c534" (UID: "342a2378-e673-42df-bc30-91af7ba1c534"). InnerVolumeSpecName "kube-api-access-v8xjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:27:16.476545 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:16.476501 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/342a2378-e673-42df-bc30-91af7ba1c534-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:16.476545 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:16.476539 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8xjl\" (UniqueName: \"kubernetes.io/projected/342a2378-e673-42df-bc30-91af7ba1c534-kube-api-access-v8xjl\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:16.476545 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:16.476553 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/342a2378-e673-42df-bc30-91af7ba1c534-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:17.285873 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:17.285834 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d" Nov 03 21:27:17.442947 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:17.442910 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d"] Nov 03 21:27:17.489823 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:17.489779 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1dda2c-b8fc5b854-vdw7d"] Nov 03 21:27:17.651334 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:17.651247 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="342a2378-e673-42df-bc30-91af7ba1c534" path="/var/lib/kubelet/pods/342a2378-e673-42df-bc30-91af7ba1c534/volumes" Nov 03 21:27:22.878912 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:22.878874 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx"] Nov 03 21:27:22.879520 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:22.879504 5615 cpu_manager.go:414] "containerMap: removing container" podUID="342a2378-e673-42df-bc30-91af7ba1c534" containerName="maas-api" Nov 03 21:27:22.879582 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:22.879522 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="342a2378-e673-42df-bc30-91af7ba1c534" containerName="maas-api" Nov 03 21:27:22.879647 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:22.879636 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="342a2378-e673-42df-bc30-91af7ba1c534" containerName="maas-api" Nov 03 21:27:22.883420 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:22.883394 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:22.970364 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:22.970321 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:22.971456 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:22.971428 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:22.971722 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:22.971703 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:22.971883 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:22.971867 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:22.972045 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:22.972030 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:22.972243 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:22.972226 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dswlw\" (UniqueName: \"kubernetes.io/projected/d837c860-39f5-4cf0-b6f3-ab441ee9e580-kube-api-access-dswlw\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:22.972413 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:22.972398 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:22.972566 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:22.972551 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:22.972779 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:22.972759 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:22.992655 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:22.992601 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-e84357-openshift-default-dockercfg-t4w57\"" Nov 03 21:27:22.993170 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:22.993090 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx"] Nov 03 21:27:23.073980 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:23.073943 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:23.074204 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:23.073991 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:23.074204 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:23.074033 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:23.074204 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:23.074115 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dswlw\" (UniqueName: \"kubernetes.io/projected/d837c860-39f5-4cf0-b6f3-ab441ee9e580-kube-api-access-dswlw\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:23.074204 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:23.074164 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:23.074204 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:23.074202 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:23.074479 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:23.074265 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:23.074479 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:23.074411 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:23.074479 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:23.074438 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:23.074648 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:23.074490 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:23.074648 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:23.074517 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:23.074648 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:23.074574 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:23.074801 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:23.074765 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:23.074856 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:23.074818 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:23.076897 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:23.076863 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:23.077014 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:23.076992 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:23.132771 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:23.132687 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:23.132934 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:23.132876 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dswlw\" (UniqueName: \"kubernetes.io/projected/d837c860-39f5-4cf0-b6f3-ab441ee9e580-kube-api-access-dswlw\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:23.198020 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:23.197977 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:23.624561 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:23.624531 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx"] Nov 03 21:27:23.626618 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:27:23.626579 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd837c860_39f5_4cf0_b6f3_ab441ee9e580.slice/crio-fbe7e66dc48361459171b6f50ce9617ec0e602d2650be3eee499d6210b3b44d9 WatchSource:0}: Error finding container fbe7e66dc48361459171b6f50ce9617ec0e602d2650be3eee499d6210b3b44d9: Status 404 returned error can't find the container with id fbe7e66dc48361459171b6f50ce9617ec0e602d2650be3eee499d6210b3b44d9 Nov 03 21:27:23.629357 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:23.629322 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:27:23.629427 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:23.629393 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:27:23.629427 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:23.629422 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:27:24.335505 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:24.335465 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" event={"ID":"d837c860-39f5-4cf0-b6f3-ab441ee9e580","Type":"ContainerStarted","Data":"01142474b9e27ab0d5602c198f34c83574c5124f788b7696a9cd1535cc904727"} Nov 03 21:27:24.335944 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:24.335551 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" event={"ID":"d837c860-39f5-4cf0-b6f3-ab441ee9e580","Type":"ContainerStarted","Data":"fbe7e66dc48361459171b6f50ce9617ec0e602d2650be3eee499d6210b3b44d9"} Nov 03 21:27:24.497943 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:24.497876 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" podStartSLOduration=2.497855327 podStartE2EDuration="2.497855327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:27:24.496390166 +0000 UTC m=+2338.964168891" watchObservedRunningTime="2026-11-03 21:27:24.497855327 +0000 UTC m=+2338.965634051" Nov 03 21:27:25.216110 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:25.213313 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:25.216110 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:25.213472 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 21:27:25.216110 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:25.213531 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" podUID="d837c860-39f5-4cf0-b6f3-ab441ee9e580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 21:27:26.218066 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:26.217596 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 21:27:26.218066 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:26.217668 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" podUID="d837c860-39f5-4cf0-b6f3-ab441ee9e580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 21:27:27.217156 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:27.216529 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 21:27:27.217156 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:27.216594 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" podUID="d837c860-39f5-4cf0-b6f3-ab441ee9e580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 21:27:28.202099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:28.201475 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 21:27:28.202099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:28.201538 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" podUID="d837c860-39f5-4cf0-b6f3-ab441ee9e580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 21:27:29.220223 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:29.220179 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 21:27:29.220820 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:29.220787 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" podUID="d837c860-39f5-4cf0-b6f3-ab441ee9e580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 21:27:30.199365 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:30.199321 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 21:27:30.199633 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:30.199410 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" podUID="d837c860-39f5-4cf0-b6f3-ab441ee9e580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 21:27:31.203855 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:31.203824 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:31.204304 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:31.204245 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:31.204902 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:27:31.204883 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:27:40.619617 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:27:40.619532 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:28:09.850798 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:09.850758 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk"] Nov 03 21:28:09.851388 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:09.850969 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk" podUID="ab30a7f6-78c8-4365-b2d5-7040cac18697" containerName="payload-processing" containerID="cri-o://afa04589846cf9b4f8e7008151ee233c94e36f87f86d05d2d340bda85c761c64" gracePeriod=30 Nov 03 21:28:09.899660 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:09.899623 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-7d548d4ccd-vfgnw"] Nov 03 21:28:09.899879 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:09.899837 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-7d548d4ccd-vfgnw" podUID="f7915873-20d0-4151-937d-11581c1a1104" containerName="payload-pre-processing" containerID="cri-o://08073f5701f4fe9f985fa8dd85e0f93266c440ec26c5e0542caf26077ded4606" gracePeriod=30 Nov 03 21:28:10.177898 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.177872 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk" Nov 03 21:28:10.195893 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.195867 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-7d548d4ccd-vfgnw" Nov 03 21:28:10.304412 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.304378 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7915873-20d0-4151-937d-11581c1a1104-kube-api-access-sjmnq\" (UniqueName: \"kubernetes.io/projected/f7915873-20d0-4151-937d-11581c1a1104-kube-api-access-sjmnq\") pod \"f7915873-20d0-4151-937d-11581c1a1104\" (UID: \"f7915873-20d0-4151-937d-11581c1a1104\") " Nov 03 21:28:10.304661 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.304431 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab30a7f6-78c8-4365-b2d5-7040cac18697-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab30a7f6-78c8-4365-b2d5-7040cac18697-plugins-config-volume\") pod \"ab30a7f6-78c8-4365-b2d5-7040cac18697\" (UID: \"ab30a7f6-78c8-4365-b2d5-7040cac18697\") " Nov 03 21:28:10.304661 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.304556 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7915873-20d0-4151-937d-11581c1a1104-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7915873-20d0-4151-937d-11581c1a1104-plugins-config-volume\") pod \"f7915873-20d0-4151-937d-11581c1a1104\" (UID: \"f7915873-20d0-4151-937d-11581c1a1104\") " Nov 03 21:28:10.304661 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.304601 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab30a7f6-78c8-4365-b2d5-7040cac18697-kube-api-access-szp72\" (UniqueName: \"kubernetes.io/projected/ab30a7f6-78c8-4365-b2d5-7040cac18697-kube-api-access-szp72\") pod \"ab30a7f6-78c8-4365-b2d5-7040cac18697\" (UID: \"ab30a7f6-78c8-4365-b2d5-7040cac18697\") " Nov 03 21:28:10.306496 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.306461 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab30a7f6-78c8-4365-b2d5-7040cac18697-plugins-config-volume" pod "ab30a7f6-78c8-4365-b2d5-7040cac18697" (UID: "ab30a7f6-78c8-4365-b2d5-7040cac18697"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:28:10.307016 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.306983 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7915873-20d0-4151-937d-11581c1a1104-plugins-config-volume" pod "f7915873-20d0-4151-937d-11581c1a1104" (UID: "f7915873-20d0-4151-937d-11581c1a1104"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:28:10.308803 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.308766 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab30a7f6-78c8-4365-b2d5-7040cac18697-kube-api-access-szp72" pod "ab30a7f6-78c8-4365-b2d5-7040cac18697" (UID: "ab30a7f6-78c8-4365-b2d5-7040cac18697"). InnerVolumeSpecName "kube-api-access-szp72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:28:10.309363 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.309336 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7915873-20d0-4151-937d-11581c1a1104-kube-api-access-sjmnq" pod "f7915873-20d0-4151-937d-11581c1a1104" (UID: "f7915873-20d0-4151-937d-11581c1a1104"). InnerVolumeSpecName "kube-api-access-sjmnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:28:10.407476 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.407379 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7915873-20d0-4151-937d-11581c1a1104-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:10.407476 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.407417 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szp72\" (UniqueName: \"kubernetes.io/projected/ab30a7f6-78c8-4365-b2d5-7040cac18697-kube-api-access-szp72\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:10.407476 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.407431 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjmnq\" (UniqueName: \"kubernetes.io/projected/f7915873-20d0-4151-937d-11581c1a1104-kube-api-access-sjmnq\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:10.407476 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.407443 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab30a7f6-78c8-4365-b2d5-7040cac18697-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:10.617602 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.617545 5615 generic.go:356] "Generic (PLEG): container finished" podID="f7915873-20d0-4151-937d-11581c1a1104" containerID="08073f5701f4fe9f985fa8dd85e0f93266c440ec26c5e0542caf26077ded4606" exitCode=0 Nov 03 21:28:10.617789 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.617614 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-7d548d4ccd-vfgnw" Nov 03 21:28:10.617789 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.617641 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-7d548d4ccd-vfgnw" event={"ID":"f7915873-20d0-4151-937d-11581c1a1104","Type":"ContainerDied","Data":"08073f5701f4fe9f985fa8dd85e0f93266c440ec26c5e0542caf26077ded4606"} Nov 03 21:28:10.617789 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.617707 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-7d548d4ccd-vfgnw" event={"ID":"f7915873-20d0-4151-937d-11581c1a1104","Type":"ContainerDied","Data":"c487214a7174e6e40fbd59b964cafad5d34b70bba597e6ab52658ab259b6cacc"} Nov 03 21:28:10.617789 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.617738 5615 scope.go:122] "RemoveContainer" containerID="08073f5701f4fe9f985fa8dd85e0f93266c440ec26c5e0542caf26077ded4606" Nov 03 21:28:10.619536 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.619512 5615 generic.go:356] "Generic (PLEG): container finished" podID="ab30a7f6-78c8-4365-b2d5-7040cac18697" containerID="afa04589846cf9b4f8e7008151ee233c94e36f87f86d05d2d340bda85c761c64" exitCode=0 Nov 03 21:28:10.619653 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.619567 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk" Nov 03 21:28:10.619653 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.619596 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk" event={"ID":"ab30a7f6-78c8-4365-b2d5-7040cac18697","Type":"ContainerDied","Data":"afa04589846cf9b4f8e7008151ee233c94e36f87f86d05d2d340bda85c761c64"} Nov 03 21:28:10.619653 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.619640 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk" event={"ID":"ab30a7f6-78c8-4365-b2d5-7040cac18697","Type":"ContainerDied","Data":"2a82aaea051f48308fdd54212ab58c791c838995046cc1b2248119d03a9c8707"} Nov 03 21:28:10.623503 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:28:10.623471 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:28:10.629509 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.629487 5615 scope.go:122] "RemoveContainer" containerID="08073f5701f4fe9f985fa8dd85e0f93266c440ec26c5e0542caf26077ded4606" Nov 03 21:28:10.629817 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:28:10.629794 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08073f5701f4fe9f985fa8dd85e0f93266c440ec26c5e0542caf26077ded4606\": container with ID starting with 08073f5701f4fe9f985fa8dd85e0f93266c440ec26c5e0542caf26077ded4606 not found: ID does not exist" containerID="08073f5701f4fe9f985fa8dd85e0f93266c440ec26c5e0542caf26077ded4606" Nov 03 21:28:10.629920 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.629824 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08073f5701f4fe9f985fa8dd85e0f93266c440ec26c5e0542caf26077ded4606"} err="failed to get container status \"08073f5701f4fe9f985fa8dd85e0f93266c440ec26c5e0542caf26077ded4606\": rpc error: code = NotFound desc = could not find container \"08073f5701f4fe9f985fa8dd85e0f93266c440ec26c5e0542caf26077ded4606\": container with ID starting with 08073f5701f4fe9f985fa8dd85e0f93266c440ec26c5e0542caf26077ded4606 not found: ID does not exist" Nov 03 21:28:10.629920 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.629847 5615 scope.go:122] "RemoveContainer" containerID="afa04589846cf9b4f8e7008151ee233c94e36f87f86d05d2d340bda85c761c64" Nov 03 21:28:10.639138 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.639097 5615 scope.go:122] "RemoveContainer" containerID="afa04589846cf9b4f8e7008151ee233c94e36f87f86d05d2d340bda85c761c64" Nov 03 21:28:10.639357 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:28:10.639339 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afa04589846cf9b4f8e7008151ee233c94e36f87f86d05d2d340bda85c761c64\": container with ID starting with afa04589846cf9b4f8e7008151ee233c94e36f87f86d05d2d340bda85c761c64 not found: ID does not exist" containerID="afa04589846cf9b4f8e7008151ee233c94e36f87f86d05d2d340bda85c761c64" Nov 03 21:28:10.639403 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.639366 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afa04589846cf9b4f8e7008151ee233c94e36f87f86d05d2d340bda85c761c64"} err="failed to get container status \"afa04589846cf9b4f8e7008151ee233c94e36f87f86d05d2d340bda85c761c64\": rpc error: code = NotFound desc = could not find container \"afa04589846cf9b4f8e7008151ee233c94e36f87f86d05d2d340bda85c761c64\": container with ID starting with afa04589846cf9b4f8e7008151ee233c94e36f87f86d05d2d340bda85c761c64 not found: ID does not exist" Nov 03 21:28:10.774248 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.774213 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk"] Nov 03 21:28:10.778332 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.778292 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eae138-554549c6b8-l4msk"] Nov 03 21:28:10.882705 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.882670 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-7d548d4ccd-vfgnw"] Nov 03 21:28:10.886558 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:10.886529 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-7d548d4ccd-vfgnw"] Nov 03 21:28:11.653531 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:11.653494 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab30a7f6-78c8-4365-b2d5-7040cac18697" path="/var/lib/kubelet/pods/ab30a7f6-78c8-4365-b2d5-7040cac18697/volumes" Nov 03 21:28:11.654138 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:11.654115 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7915873-20d0-4151-937d-11581c1a1104" path="/var/lib/kubelet/pods/f7915873-20d0-4151-937d-11581c1a1104/volumes" Nov 03 21:28:11.677997 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:11.677961 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7"] Nov 03 21:28:11.678265 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:11.678237 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" podUID="f82eafd0-9398-4cec-9b95-e510391864ab" containerName="istio-proxy" containerID="cri-o://bc73b1363a507690decd4d5ae2d0c21653d49b7b199a7a0b3d31c4e4a9518b67" gracePeriod=30 Nov 03 21:28:16.950142 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:16.950066 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:28:16.992342 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:16.992303 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f82eafd0-9398-4cec-9b95-e510391864ab-kube-api-access-r98pf\" (UniqueName: \"kubernetes.io/projected/f82eafd0-9398-4cec-9b95-e510391864ab-kube-api-access-r98pf\") pod \"f82eafd0-9398-4cec-9b95-e510391864ab\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " Nov 03 21:28:16.992532 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:16.992395 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f82eafd0-9398-4cec-9b95-e510391864ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f82eafd0-9398-4cec-9b95-e510391864ab-istio-podinfo\") pod \"f82eafd0-9398-4cec-9b95-e510391864ab\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " Nov 03 21:28:16.992532 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:16.992424 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-workload-socket\") pod \"f82eafd0-9398-4cec-9b95-e510391864ab\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " Nov 03 21:28:16.992532 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:16.992471 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-workload-certs\") pod \"f82eafd0-9398-4cec-9b95-e510391864ab\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " Nov 03 21:28:16.992532 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:16.992491 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-istio-envoy\") pod \"f82eafd0-9398-4cec-9b95-e510391864ab\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " Nov 03 21:28:16.992532 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:16.992511 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f82eafd0-9398-4cec-9b95-e510391864ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f82eafd0-9398-4cec-9b95-e510391864ab-istiod-ca-cert\") pod \"f82eafd0-9398-4cec-9b95-e510391864ab\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " Nov 03 21:28:16.992794 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:16.992560 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-istio-data\") pod \"f82eafd0-9398-4cec-9b95-e510391864ab\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " Nov 03 21:28:16.992794 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:16.992640 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-credential-socket\") pod \"f82eafd0-9398-4cec-9b95-e510391864ab\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " Nov 03 21:28:16.992794 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:16.992728 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f82eafd0-9398-4cec-9b95-e510391864ab-istio-token\" (UniqueName: \"kubernetes.io/projected/f82eafd0-9398-4cec-9b95-e510391864ab-istio-token\") pod \"f82eafd0-9398-4cec-9b95-e510391864ab\" (UID: \"f82eafd0-9398-4cec-9b95-e510391864ab\") " Nov 03 21:28:16.992949 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:16.992919 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-workload-socket" pod "f82eafd0-9398-4cec-9b95-e510391864ab" (UID: "f82eafd0-9398-4cec-9b95-e510391864ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:28:16.992949 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:16.992925 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-istio-data" pod "f82eafd0-9398-4cec-9b95-e510391864ab" (UID: "f82eafd0-9398-4cec-9b95-e510391864ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:28:16.993060 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:16.992956 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-workload-certs" pod "f82eafd0-9398-4cec-9b95-e510391864ab" (UID: "f82eafd0-9398-4cec-9b95-e510391864ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:28:16.993060 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:16.993004 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f82eafd0-9398-4cec-9b95-e510391864ab-istiod-ca-cert" pod "f82eafd0-9398-4cec-9b95-e510391864ab" (UID: "f82eafd0-9398-4cec-9b95-e510391864ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:28:16.993202 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:16.993128 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-credential-socket" pod "f82eafd0-9398-4cec-9b95-e510391864ab" (UID: "f82eafd0-9398-4cec-9b95-e510391864ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:28:16.993479 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:16.993456 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:16.993564 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:16.993485 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:16.993564 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:16.993502 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f82eafd0-9398-4cec-9b95-e510391864ab-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:16.993564 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:16.993520 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:16.993564 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:16.993535 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:16.994787 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:16.994759 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f82eafd0-9398-4cec-9b95-e510391864ab-kube-api-access-r98pf" pod "f82eafd0-9398-4cec-9b95-e510391864ab" (UID: "f82eafd0-9398-4cec-9b95-e510391864ab"). InnerVolumeSpecName "kube-api-access-r98pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:28:16.994787 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:16.994766 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f82eafd0-9398-4cec-9b95-e510391864ab-istio-podinfo" pod "f82eafd0-9398-4cec-9b95-e510391864ab" (UID: "f82eafd0-9398-4cec-9b95-e510391864ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:28:16.994964 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:16.994931 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f82eafd0-9398-4cec-9b95-e510391864ab-istio-token" pod "f82eafd0-9398-4cec-9b95-e510391864ab" (UID: "f82eafd0-9398-4cec-9b95-e510391864ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:28:16.995114 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:16.995063 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-istio-envoy" pod "f82eafd0-9398-4cec-9b95-e510391864ab" (UID: "f82eafd0-9398-4cec-9b95-e510391864ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:28:17.094624 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:17.094586 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f82eafd0-9398-4cec-9b95-e510391864ab-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:17.094624 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:17.094624 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r98pf\" (UniqueName: \"kubernetes.io/projected/f82eafd0-9398-4cec-9b95-e510391864ab-kube-api-access-r98pf\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:17.094834 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:17.094639 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f82eafd0-9398-4cec-9b95-e510391864ab-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:17.094834 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:17.094652 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f82eafd0-9398-4cec-9b95-e510391864ab-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:17.666170 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:17.666134 5615 generic.go:356] "Generic (PLEG): container finished" podID="f82eafd0-9398-4cec-9b95-e510391864ab" containerID="bc73b1363a507690decd4d5ae2d0c21653d49b7b199a7a0b3d31c4e4a9518b67" exitCode=0 Nov 03 21:28:17.666315 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:17.666187 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" Nov 03 21:28:17.666315 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:17.666210 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" event={"ID":"f82eafd0-9398-4cec-9b95-e510391864ab","Type":"ContainerDied","Data":"bc73b1363a507690decd4d5ae2d0c21653d49b7b199a7a0b3d31c4e4a9518b67"} Nov 03 21:28:17.666315 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:17.666268 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7" event={"ID":"f82eafd0-9398-4cec-9b95-e510391864ab","Type":"ContainerDied","Data":"3ff87e4abe85f6040d609e7e1fd30ec03b320d248af20e0346342a70c5756a39"} Nov 03 21:28:17.666315 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:17.666291 5615 scope.go:122] "RemoveContainer" containerID="bc73b1363a507690decd4d5ae2d0c21653d49b7b199a7a0b3d31c4e4a9518b67" Nov 03 21:28:17.676913 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:17.676890 5615 scope.go:122] "RemoveContainer" containerID="efc7d29bc549dd372ddcb388afb5e8cadc82bf8e97142c1f562db32ff49fe629" Nov 03 21:28:17.686485 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:17.686455 5615 scope.go:122] "RemoveContainer" containerID="bc73b1363a507690decd4d5ae2d0c21653d49b7b199a7a0b3d31c4e4a9518b67" Nov 03 21:28:17.686857 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:28:17.686825 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc73b1363a507690decd4d5ae2d0c21653d49b7b199a7a0b3d31c4e4a9518b67\": container with ID starting with bc73b1363a507690decd4d5ae2d0c21653d49b7b199a7a0b3d31c4e4a9518b67 not found: ID does not exist" containerID="bc73b1363a507690decd4d5ae2d0c21653d49b7b199a7a0b3d31c4e4a9518b67" Nov 03 21:28:17.687001 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:17.686866 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc73b1363a507690decd4d5ae2d0c21653d49b7b199a7a0b3d31c4e4a9518b67"} err="failed to get container status \"bc73b1363a507690decd4d5ae2d0c21653d49b7b199a7a0b3d31c4e4a9518b67\": rpc error: code = NotFound desc = could not find container \"bc73b1363a507690decd4d5ae2d0c21653d49b7b199a7a0b3d31c4e4a9518b67\": container with ID starting with bc73b1363a507690decd4d5ae2d0c21653d49b7b199a7a0b3d31c4e4a9518b67 not found: ID does not exist" Nov 03 21:28:17.687001 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:17.686890 5615 scope.go:122] "RemoveContainer" containerID="efc7d29bc549dd372ddcb388afb5e8cadc82bf8e97142c1f562db32ff49fe629" Nov 03 21:28:17.687291 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:28:17.687248 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efc7d29bc549dd372ddcb388afb5e8cadc82bf8e97142c1f562db32ff49fe629\": container with ID starting with efc7d29bc549dd372ddcb388afb5e8cadc82bf8e97142c1f562db32ff49fe629 not found: ID does not exist" containerID="efc7d29bc549dd372ddcb388afb5e8cadc82bf8e97142c1f562db32ff49fe629" Nov 03 21:28:17.687291 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:17.687280 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc7d29bc549dd372ddcb388afb5e8cadc82bf8e97142c1f562db32ff49fe629"} err="failed to get container status \"efc7d29bc549dd372ddcb388afb5e8cadc82bf8e97142c1f562db32ff49fe629\": rpc error: code = NotFound desc = could not find container \"efc7d29bc549dd372ddcb388afb5e8cadc82bf8e97142c1f562db32ff49fe629\": container with ID starting with efc7d29bc549dd372ddcb388afb5e8cadc82bf8e97142c1f562db32ff49fe629 not found: ID does not exist" Nov 03 21:28:17.870907 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:17.870870 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7"] Nov 03 21:28:17.920040 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:17.919938 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-eae138-openshift-default-5fd596c665-rlxj7"] Nov 03 21:28:19.651621 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:19.651586 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f82eafd0-9398-4cec-9b95-e510391864ab" path="/var/lib/kubelet/pods/f82eafd0-9398-4cec-9b95-e510391864ab/volumes" Nov 03 21:28:20.147569 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.147535 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq"] Nov 03 21:28:20.147760 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.147722 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq" podUID="922f3c9b-ce05-4037-8b8c-b229f495e393" containerName="maas-api" containerID="cri-o://e29e311d06d97edbfabfed05be64a810db5999c8bfc4e54cb913053cc67231cc" gracePeriod=30 Nov 03 21:28:20.437602 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.437571 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq" Nov 03 21:28:20.564130 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.556999 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/922f3c9b-ce05-4037-8b8c-b229f495e393-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/922f3c9b-ce05-4037-8b8c-b229f495e393-maas-api-tls\") pod \"922f3c9b-ce05-4037-8b8c-b229f495e393\" (UID: \"922f3c9b-ce05-4037-8b8c-b229f495e393\") " Nov 03 21:28:20.564130 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.557092 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/922f3c9b-ce05-4037-8b8c-b229f495e393-metrics-certs\" (UniqueName: \"kubernetes.io/secret/922f3c9b-ce05-4037-8b8c-b229f495e393-metrics-certs\") pod \"922f3c9b-ce05-4037-8b8c-b229f495e393\" (UID: \"922f3c9b-ce05-4037-8b8c-b229f495e393\") " Nov 03 21:28:20.564130 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.557162 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/922f3c9b-ce05-4037-8b8c-b229f495e393-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/922f3c9b-ce05-4037-8b8c-b229f495e393-openshift-service-ca\") pod \"922f3c9b-ce05-4037-8b8c-b229f495e393\" (UID: \"922f3c9b-ce05-4037-8b8c-b229f495e393\") " Nov 03 21:28:20.564130 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.557226 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/922f3c9b-ce05-4037-8b8c-b229f495e393-kube-api-access-h6266\" (UniqueName: \"kubernetes.io/projected/922f3c9b-ce05-4037-8b8c-b229f495e393-kube-api-access-h6266\") pod \"922f3c9b-ce05-4037-8b8c-b229f495e393\" (UID: \"922f3c9b-ce05-4037-8b8c-b229f495e393\") " Nov 03 21:28:20.574679 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.574622 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/922f3c9b-ce05-4037-8b8c-b229f495e393-openshift-service-ca" pod "922f3c9b-ce05-4037-8b8c-b229f495e393" (UID: "922f3c9b-ce05-4037-8b8c-b229f495e393"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:28:20.592244 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.591265 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922f3c9b-ce05-4037-8b8c-b229f495e393-kube-api-access-h6266" pod "922f3c9b-ce05-4037-8b8c-b229f495e393" (UID: "922f3c9b-ce05-4037-8b8c-b229f495e393"). InnerVolumeSpecName "kube-api-access-h6266". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:28:20.592732 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.592685 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/922f3c9b-ce05-4037-8b8c-b229f495e393-metrics-certs" pod "922f3c9b-ce05-4037-8b8c-b229f495e393" (UID: "922f3c9b-ce05-4037-8b8c-b229f495e393"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:28:20.593066 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.592890 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/922f3c9b-ce05-4037-8b8c-b229f495e393-maas-api-tls" pod "922f3c9b-ce05-4037-8b8c-b229f495e393" (UID: "922f3c9b-ce05-4037-8b8c-b229f495e393"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:28:20.614873 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.614543 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj"] Nov 03 21:28:20.614873 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.614802 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj" podUID="cd915ca2-fb6f-4a5b-bb22-f281df8f7337" containerName="payload-processing" containerID="cri-o://bfcedbe952a40e740fb7316c71c1c1e0c8cbb5081966afc514e1a268f9e2090e" gracePeriod=30 Nov 03 21:28:20.658338 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.658298 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/922f3c9b-ce05-4037-8b8c-b229f495e393-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:20.658338 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.658331 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/922f3c9b-ce05-4037-8b8c-b229f495e393-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:20.658338 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.658342 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/922f3c9b-ce05-4037-8b8c-b229f495e393-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:20.658774 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.658352 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6266\" (UniqueName: \"kubernetes.io/projected/922f3c9b-ce05-4037-8b8c-b229f495e393-kube-api-access-h6266\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:20.689598 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.689500 5615 generic.go:356] "Generic (PLEG): container finished" podID="922f3c9b-ce05-4037-8b8c-b229f495e393" containerID="e29e311d06d97edbfabfed05be64a810db5999c8bfc4e54cb913053cc67231cc" exitCode=0 Nov 03 21:28:20.689598 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.689547 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq" Nov 03 21:28:20.689598 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.689585 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq" event={"ID":"922f3c9b-ce05-4037-8b8c-b229f495e393","Type":"ContainerDied","Data":"e29e311d06d97edbfabfed05be64a810db5999c8bfc4e54cb913053cc67231cc"} Nov 03 21:28:20.689875 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.689650 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq" event={"ID":"922f3c9b-ce05-4037-8b8c-b229f495e393","Type":"ContainerDied","Data":"11a58c57b1092942698cb7880bab674da8987a34818b81a682ce183bce50ea3d"} Nov 03 21:28:20.689875 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.689682 5615 scope.go:122] "RemoveContainer" containerID="e29e311d06d97edbfabfed05be64a810db5999c8bfc4e54cb913053cc67231cc" Nov 03 21:28:20.730384 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.730357 5615 scope.go:122] "RemoveContainer" containerID="e29e311d06d97edbfabfed05be64a810db5999c8bfc4e54cb913053cc67231cc" Nov 03 21:28:20.730773 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:28:20.730749 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e29e311d06d97edbfabfed05be64a810db5999c8bfc4e54cb913053cc67231cc\": container with ID starting with e29e311d06d97edbfabfed05be64a810db5999c8bfc4e54cb913053cc67231cc not found: ID does not exist" containerID="e29e311d06d97edbfabfed05be64a810db5999c8bfc4e54cb913053cc67231cc" Nov 03 21:28:20.730886 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.730784 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e29e311d06d97edbfabfed05be64a810db5999c8bfc4e54cb913053cc67231cc"} err="failed to get container status \"e29e311d06d97edbfabfed05be64a810db5999c8bfc4e54cb913053cc67231cc\": rpc error: code = NotFound desc = could not find container \"e29e311d06d97edbfabfed05be64a810db5999c8bfc4e54cb913053cc67231cc\": container with ID starting with e29e311d06d97edbfabfed05be64a810db5999c8bfc4e54cb913053cc67231cc not found: ID does not exist" Nov 03 21:28:20.839591 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.839548 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq"] Nov 03 21:28:20.841622 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.841594 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a77060-5f96c89d5b-9rklq"] Nov 03 21:28:20.880814 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.880787 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj" Nov 03 21:28:20.963335 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.963240 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd915ca2-fb6f-4a5b-bb22-f281df8f7337-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd915ca2-fb6f-4a5b-bb22-f281df8f7337-plugins-config-volume\") pod \"cd915ca2-fb6f-4a5b-bb22-f281df8f7337\" (UID: \"cd915ca2-fb6f-4a5b-bb22-f281df8f7337\") " Nov 03 21:28:20.963512 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.963392 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd915ca2-fb6f-4a5b-bb22-f281df8f7337-kube-api-access-qq5md\" (UniqueName: \"kubernetes.io/projected/cd915ca2-fb6f-4a5b-bb22-f281df8f7337-kube-api-access-qq5md\") pod \"cd915ca2-fb6f-4a5b-bb22-f281df8f7337\" (UID: \"cd915ca2-fb6f-4a5b-bb22-f281df8f7337\") " Nov 03 21:28:20.963737 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.963706 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd915ca2-fb6f-4a5b-bb22-f281df8f7337-plugins-config-volume" pod "cd915ca2-fb6f-4a5b-bb22-f281df8f7337" (UID: "cd915ca2-fb6f-4a5b-bb22-f281df8f7337"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:28:20.963900 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.963880 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd915ca2-fb6f-4a5b-bb22-f281df8f7337-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:20.965620 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:20.965593 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd915ca2-fb6f-4a5b-bb22-f281df8f7337-kube-api-access-qq5md" pod "cd915ca2-fb6f-4a5b-bb22-f281df8f7337" (UID: "cd915ca2-fb6f-4a5b-bb22-f281df8f7337"). InnerVolumeSpecName "kube-api-access-qq5md". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:28:21.077194 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:21.074242 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq5md\" (UniqueName: \"kubernetes.io/projected/cd915ca2-fb6f-4a5b-bb22-f281df8f7337-kube-api-access-qq5md\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:21.651972 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:21.651939 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="922f3c9b-ce05-4037-8b8c-b229f495e393" path="/var/lib/kubelet/pods/922f3c9b-ce05-4037-8b8c-b229f495e393/volumes" Nov 03 21:28:21.697133 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:21.697097 5615 generic.go:356] "Generic (PLEG): container finished" podID="cd915ca2-fb6f-4a5b-bb22-f281df8f7337" containerID="bfcedbe952a40e740fb7316c71c1c1e0c8cbb5081966afc514e1a268f9e2090e" exitCode=0 Nov 03 21:28:21.697611 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:21.697153 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj" Nov 03 21:28:21.697611 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:21.697190 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj" event={"ID":"cd915ca2-fb6f-4a5b-bb22-f281df8f7337","Type":"ContainerDied","Data":"bfcedbe952a40e740fb7316c71c1c1e0c8cbb5081966afc514e1a268f9e2090e"} Nov 03 21:28:21.697611 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:21.697251 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj" event={"ID":"cd915ca2-fb6f-4a5b-bb22-f281df8f7337","Type":"ContainerDied","Data":"556cb1f34ed37668579ffd0e355f92124b38dbc700d733e9c7ab7fdc646be2cb"} Nov 03 21:28:21.697611 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:21.697277 5615 scope.go:122] "RemoveContainer" containerID="bfcedbe952a40e740fb7316c71c1c1e0c8cbb5081966afc514e1a268f9e2090e" Nov 03 21:28:21.706827 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:21.706802 5615 scope.go:122] "RemoveContainer" containerID="bfcedbe952a40e740fb7316c71c1c1e0c8cbb5081966afc514e1a268f9e2090e" Nov 03 21:28:21.707198 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:28:21.707171 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfcedbe952a40e740fb7316c71c1c1e0c8cbb5081966afc514e1a268f9e2090e\": container with ID starting with bfcedbe952a40e740fb7316c71c1c1e0c8cbb5081966afc514e1a268f9e2090e not found: ID does not exist" containerID="bfcedbe952a40e740fb7316c71c1c1e0c8cbb5081966afc514e1a268f9e2090e" Nov 03 21:28:21.707267 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:21.707207 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfcedbe952a40e740fb7316c71c1c1e0c8cbb5081966afc514e1a268f9e2090e"} err="failed to get container status \"bfcedbe952a40e740fb7316c71c1c1e0c8cbb5081966afc514e1a268f9e2090e\": rpc error: code = NotFound desc = could not find container \"bfcedbe952a40e740fb7316c71c1c1e0c8cbb5081966afc514e1a268f9e2090e\": container with ID starting with bfcedbe952a40e740fb7316c71c1c1e0c8cbb5081966afc514e1a268f9e2090e not found: ID does not exist" Nov 03 21:28:21.853197 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:21.853153 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj"] Nov 03 21:28:21.857922 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:21.857895 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a77060-744f96685d-z96kj"] Nov 03 21:28:23.651780 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:23.651743 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd915ca2-fb6f-4a5b-bb22-f281df8f7337" path="/var/lib/kubelet/pods/cd915ca2-fb6f-4a5b-bb22-f281df8f7337/volumes" Nov 03 21:28:26.456761 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:26.456722 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj"] Nov 03 21:28:26.457257 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:26.456944 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" podUID="f285754f-0207-4721-a4f6-62ebfe9a2223" containerName="istio-proxy" containerID="cri-o://f3307ec28d82ecd57c62ea835bfda015ea8fea567cf815ebe499c9590ccdcb2c" gracePeriod=30 Nov 03 21:28:26.769923 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:26.769891 5615 scope.go:122] "RemoveContainer" containerID="6af3dcef8ada1fd8d46dfcc4c69138b7080a8889d1fa1e5ad988ac2efdf5748f" Nov 03 21:28:27.620725 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:27.620687 5615 patch_prober.go:29] interesting pod/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:28:27.621117 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:27.620756 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" podUID="f285754f-0207-4721-a4f6-62ebfe9a2223" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:28:28.526892 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.526853 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc"] Nov 03 21:28:28.527404 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.527388 5615 cpu_manager.go:414] "containerMap: removing container" podUID="f82eafd0-9398-4cec-9b95-e510391864ab" containerName="istio-proxy" Nov 03 21:28:28.527460 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.527405 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f82eafd0-9398-4cec-9b95-e510391864ab" containerName="istio-proxy" Nov 03 21:28:28.527460 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.527423 5615 cpu_manager.go:414] "containerMap: removing container" podUID="cd915ca2-fb6f-4a5b-bb22-f281df8f7337" containerName="payload-processing" Nov 03 21:28:28.527460 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.527428 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd915ca2-fb6f-4a5b-bb22-f281df8f7337" containerName="payload-processing" Nov 03 21:28:28.527460 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.527436 5615 cpu_manager.go:414] "containerMap: removing container" podUID="922f3c9b-ce05-4037-8b8c-b229f495e393" containerName="maas-api" Nov 03 21:28:28.527460 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.527441 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="922f3c9b-ce05-4037-8b8c-b229f495e393" containerName="maas-api" Nov 03 21:28:28.527460 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.527453 5615 cpu_manager.go:414] "containerMap: removing container" podUID="f7915873-20d0-4151-937d-11581c1a1104" containerName="payload-pre-processing" Nov 03 21:28:28.527460 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.527457 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7915873-20d0-4151-937d-11581c1a1104" containerName="payload-pre-processing" Nov 03 21:28:28.527665 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.527468 5615 cpu_manager.go:414] "containerMap: removing container" podUID="f82eafd0-9398-4cec-9b95-e510391864ab" containerName="istio-proxy" Nov 03 21:28:28.527665 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.527472 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f82eafd0-9398-4cec-9b95-e510391864ab" containerName="istio-proxy" Nov 03 21:28:28.527665 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.527499 5615 cpu_manager.go:414] "containerMap: removing container" podUID="ab30a7f6-78c8-4365-b2d5-7040cac18697" containerName="payload-processing" Nov 03 21:28:28.527665 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.527503 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab30a7f6-78c8-4365-b2d5-7040cac18697" containerName="payload-processing" Nov 03 21:28:28.527665 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.527574 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="f82eafd0-9398-4cec-9b95-e510391864ab" containerName="istio-proxy" Nov 03 21:28:28.527665 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.527583 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd915ca2-fb6f-4a5b-bb22-f281df8f7337" containerName="payload-processing" Nov 03 21:28:28.527665 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.527589 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="922f3c9b-ce05-4037-8b8c-b229f495e393" containerName="maas-api" Nov 03 21:28:28.527665 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.527596 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7915873-20d0-4151-937d-11581c1a1104" containerName="payload-pre-processing" Nov 03 21:28:28.527665 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.527602 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="f82eafd0-9398-4cec-9b95-e510391864ab" containerName="istio-proxy" Nov 03 21:28:28.527665 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.527613 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab30a7f6-78c8-4365-b2d5-7040cac18697" containerName="payload-processing" Nov 03 21:28:28.532467 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.532442 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:28.588097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.583308 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-a41883-openshift-default-dockercfg-ps4tm\"" Nov 03 21:28:28.635222 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.635185 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc"] Nov 03 21:28:28.710156 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.709106 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-istio-envoy\") pod \"e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:28.710156 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.709262 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c842d3a-c0e4-4703-acc7-77053dd626ba-istio-podinfo\") pod \"e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:28.710156 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.709308 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4gnh\" (UniqueName: \"kubernetes.io/projected/4c842d3a-c0e4-4703-acc7-77053dd626ba-kube-api-access-m4gnh\") pod \"e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:28.710156 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.709354 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c842d3a-c0e4-4703-acc7-77053dd626ba-istiod-ca-cert\") pod \"e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:28.710156 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.709473 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-credential-socket\") pod \"e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:28.710156 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.709511 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-workload-socket\") pod \"e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:28.710156 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.709567 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-workload-certs\") pod \"e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:28.710156 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.709596 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c842d3a-c0e4-4703-acc7-77053dd626ba-istio-token\") pod \"e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:28.710156 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.709643 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-istio-data\") pod \"e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:28.811328 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.811235 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-credential-socket\") pod \"e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:28.811328 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.811290 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-workload-socket\") pod \"e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:28.811328 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.811325 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-workload-certs\") pod \"e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:28.811606 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.811344 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c842d3a-c0e4-4703-acc7-77053dd626ba-istio-token\") pod \"e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:28.811606 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.811367 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-istio-data\") pod \"e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:28.811606 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.811405 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-istio-envoy\") pod \"e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:28.811606 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.811467 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c842d3a-c0e4-4703-acc7-77053dd626ba-istio-podinfo\") pod \"e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:28.811606 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.811508 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4gnh\" (UniqueName: \"kubernetes.io/projected/4c842d3a-c0e4-4703-acc7-77053dd626ba-kube-api-access-m4gnh\") pod \"e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:28.811606 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.811553 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c842d3a-c0e4-4703-acc7-77053dd626ba-istiod-ca-cert\") pod \"e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:28.811928 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.811704 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-credential-socket\") pod \"e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:28.811928 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.811901 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-workload-socket\") pod \"e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:28.811928 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.811924 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-workload-certs\") pod \"e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:28.812236 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.812213 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-istio-data\") pod \"e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:28.812479 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.812461 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c842d3a-c0e4-4703-acc7-77053dd626ba-istiod-ca-cert\") pod \"e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:28.813945 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.813918 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-istio-envoy\") pod \"e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:28.814159 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.814141 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c842d3a-c0e4-4703-acc7-77053dd626ba-istio-podinfo\") pod \"e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:28.869679 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.869642 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c842d3a-c0e4-4703-acc7-77053dd626ba-istio-token\") pod \"e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:28.869842 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:28.869702 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4gnh\" (UniqueName: \"kubernetes.io/projected/4c842d3a-c0e4-4703-acc7-77053dd626ba-kube-api-access-m4gnh\") pod \"e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:29.145773 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:29.145678 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:29.571016 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:29.570991 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc"] Nov 03 21:28:29.573054 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:28:29.573025 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c842d3a_c0e4_4703_acc7_77053dd626ba.slice/crio-770ac3cdc709f6f083d05f3ce8c478018b4f2cd2eadb309dbd1e4462de8566c5 WatchSource:0}: Error finding container 770ac3cdc709f6f083d05f3ce8c478018b4f2cd2eadb309dbd1e4462de8566c5: Status 404 returned error can't find the container with id 770ac3cdc709f6f083d05f3ce8c478018b4f2cd2eadb309dbd1e4462de8566c5 Nov 03 21:28:29.575588 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:29.575552 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:28:29.575664 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:29.575622 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:28:29.575664 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:29.575649 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:28:29.757883 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:29.757846 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" event={"ID":"4c842d3a-c0e4-4703-acc7-77053dd626ba","Type":"ContainerStarted","Data":"b6a0b200e52325e74816d5357db26ea82cfb519572aedac14ef6e19ea27b4e3d"} Nov 03 21:28:29.758284 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:29.757906 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" event={"ID":"4c842d3a-c0e4-4703-acc7-77053dd626ba","Type":"ContainerStarted","Data":"770ac3cdc709f6f083d05f3ce8c478018b4f2cd2eadb309dbd1e4462de8566c5"} Nov 03 21:28:29.873730 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:29.873622 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" podStartSLOduration=1.873606515 podStartE2EDuration="1.873606515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:28:29.871935257 +0000 UTC m=+2404.339713981" watchObservedRunningTime="2026-11-03 21:28:29.873606515 +0000 UTC m=+2404.341385311" Nov 03 21:28:30.145898 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:30.145808 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:30.151645 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:30.151614 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:30.771288 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:30.771244 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:30.772466 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:30.772440 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:28:31.724827 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.724761 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:28:31.751809 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.751776 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f285754f-0207-4721-a4f6-62ebfe9a2223-istio-token\" (UniqueName: \"kubernetes.io/projected/f285754f-0207-4721-a4f6-62ebfe9a2223-istio-token\") pod \"f285754f-0207-4721-a4f6-62ebfe9a2223\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " Nov 03 21:28:31.751985 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.751892 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-workload-socket\") pod \"f285754f-0207-4721-a4f6-62ebfe9a2223\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " Nov 03 21:28:31.751985 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.751973 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f285754f-0207-4721-a4f6-62ebfe9a2223-kube-api-access-xd29t\" (UniqueName: \"kubernetes.io/projected/f285754f-0207-4721-a4f6-62ebfe9a2223-kube-api-access-xd29t\") pod \"f285754f-0207-4721-a4f6-62ebfe9a2223\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " Nov 03 21:28:31.752115 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.752004 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-istio-envoy\") pod \"f285754f-0207-4721-a4f6-62ebfe9a2223\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " Nov 03 21:28:31.752115 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.752035 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-credential-socket\") pod \"f285754f-0207-4721-a4f6-62ebfe9a2223\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " Nov 03 21:28:31.752232 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.752130 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-workload-certs\") pod \"f285754f-0207-4721-a4f6-62ebfe9a2223\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " Nov 03 21:28:31.752285 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.752239 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-workload-socket" pod "f285754f-0207-4721-a4f6-62ebfe9a2223" (UID: "f285754f-0207-4721-a4f6-62ebfe9a2223"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:28:31.752398 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.752370 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-credential-socket" pod "f285754f-0207-4721-a4f6-62ebfe9a2223" (UID: "f285754f-0207-4721-a4f6-62ebfe9a2223"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:28:31.752478 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.752417 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-workload-certs" pod "f285754f-0207-4721-a4f6-62ebfe9a2223" (UID: "f285754f-0207-4721-a4f6-62ebfe9a2223"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:28:31.752576 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.752554 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-istio-data\") pod \"f285754f-0207-4721-a4f6-62ebfe9a2223\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " Nov 03 21:28:31.752648 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.752599 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f285754f-0207-4721-a4f6-62ebfe9a2223-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f285754f-0207-4721-a4f6-62ebfe9a2223-istio-podinfo\") pod \"f285754f-0207-4721-a4f6-62ebfe9a2223\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " Nov 03 21:28:31.753302 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.752712 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f285754f-0207-4721-a4f6-62ebfe9a2223-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f285754f-0207-4721-a4f6-62ebfe9a2223-istiod-ca-cert\") pod \"f285754f-0207-4721-a4f6-62ebfe9a2223\" (UID: \"f285754f-0207-4721-a4f6-62ebfe9a2223\") " Nov 03 21:28:31.753302 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.752875 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-istio-data" pod "f285754f-0207-4721-a4f6-62ebfe9a2223" (UID: "f285754f-0207-4721-a4f6-62ebfe9a2223"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:28:31.753302 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.752996 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f285754f-0207-4721-a4f6-62ebfe9a2223-istiod-ca-cert" pod "f285754f-0207-4721-a4f6-62ebfe9a2223" (UID: "f285754f-0207-4721-a4f6-62ebfe9a2223"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:28:31.753302 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.753273 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f285754f-0207-4721-a4f6-62ebfe9a2223-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:31.753302 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.753297 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:31.753547 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.753311 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:31.753547 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.753324 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:31.753547 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.753337 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:31.754369 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.754346 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f285754f-0207-4721-a4f6-62ebfe9a2223-kube-api-access-xd29t" pod "f285754f-0207-4721-a4f6-62ebfe9a2223" (UID: "f285754f-0207-4721-a4f6-62ebfe9a2223"). InnerVolumeSpecName "kube-api-access-xd29t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:28:31.754497 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.754470 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f285754f-0207-4721-a4f6-62ebfe9a2223-istio-token" pod "f285754f-0207-4721-a4f6-62ebfe9a2223" (UID: "f285754f-0207-4721-a4f6-62ebfe9a2223"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:28:31.754564 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.754478 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-istio-envoy" pod "f285754f-0207-4721-a4f6-62ebfe9a2223" (UID: "f285754f-0207-4721-a4f6-62ebfe9a2223"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:28:31.755201 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.755181 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f285754f-0207-4721-a4f6-62ebfe9a2223-istio-podinfo" pod "f285754f-0207-4721-a4f6-62ebfe9a2223" (UID: "f285754f-0207-4721-a4f6-62ebfe9a2223"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:28:31.778099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.778052 5615 generic.go:356] "Generic (PLEG): container finished" podID="f285754f-0207-4721-a4f6-62ebfe9a2223" containerID="f3307ec28d82ecd57c62ea835bfda015ea8fea567cf815ebe499c9590ccdcb2c" exitCode=0 Nov 03 21:28:31.778526 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.778106 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" event={"ID":"f285754f-0207-4721-a4f6-62ebfe9a2223","Type":"ContainerDied","Data":"f3307ec28d82ecd57c62ea835bfda015ea8fea567cf815ebe499c9590ccdcb2c"} Nov 03 21:28:31.778526 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.778129 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" Nov 03 21:28:31.778526 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.778167 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj" event={"ID":"f285754f-0207-4721-a4f6-62ebfe9a2223","Type":"ContainerDied","Data":"ba6a41cf486fab1605cde6516f5fd15af75a27ce77cfffd90cedc1c508521911"} Nov 03 21:28:31.778526 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.778195 5615 scope.go:122] "RemoveContainer" containerID="f3307ec28d82ecd57c62ea835bfda015ea8fea567cf815ebe499c9590ccdcb2c" Nov 03 21:28:31.788489 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.788463 5615 scope.go:122] "RemoveContainer" containerID="cb2e92e32076d88c076279d331ee8ea131a21a8f0806c0b6eafe04c27e40fdcf" Nov 03 21:28:31.840211 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.839781 5615 scope.go:122] "RemoveContainer" containerID="f3307ec28d82ecd57c62ea835bfda015ea8fea567cf815ebe499c9590ccdcb2c" Nov 03 21:28:31.842231 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:28:31.840680 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3307ec28d82ecd57c62ea835bfda015ea8fea567cf815ebe499c9590ccdcb2c\": container with ID starting with f3307ec28d82ecd57c62ea835bfda015ea8fea567cf815ebe499c9590ccdcb2c not found: ID does not exist" containerID="f3307ec28d82ecd57c62ea835bfda015ea8fea567cf815ebe499c9590ccdcb2c" Nov 03 21:28:31.842231 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.840722 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3307ec28d82ecd57c62ea835bfda015ea8fea567cf815ebe499c9590ccdcb2c"} err="failed to get container status \"f3307ec28d82ecd57c62ea835bfda015ea8fea567cf815ebe499c9590ccdcb2c\": rpc error: code = NotFound desc = could not find container \"f3307ec28d82ecd57c62ea835bfda015ea8fea567cf815ebe499c9590ccdcb2c\": container with ID starting with f3307ec28d82ecd57c62ea835bfda015ea8fea567cf815ebe499c9590ccdcb2c not found: ID does not exist" Nov 03 21:28:31.842231 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.840746 5615 scope.go:122] "RemoveContainer" containerID="cb2e92e32076d88c076279d331ee8ea131a21a8f0806c0b6eafe04c27e40fdcf" Nov 03 21:28:31.842495 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:28:31.842316 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb2e92e32076d88c076279d331ee8ea131a21a8f0806c0b6eafe04c27e40fdcf\": container with ID starting with cb2e92e32076d88c076279d331ee8ea131a21a8f0806c0b6eafe04c27e40fdcf not found: ID does not exist" containerID="cb2e92e32076d88c076279d331ee8ea131a21a8f0806c0b6eafe04c27e40fdcf" Nov 03 21:28:31.842495 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.842352 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb2e92e32076d88c076279d331ee8ea131a21a8f0806c0b6eafe04c27e40fdcf"} err="failed to get container status \"cb2e92e32076d88c076279d331ee8ea131a21a8f0806c0b6eafe04c27e40fdcf\": rpc error: code = NotFound desc = could not find container \"cb2e92e32076d88c076279d331ee8ea131a21a8f0806c0b6eafe04c27e40fdcf\": container with ID starting with cb2e92e32076d88c076279d331ee8ea131a21a8f0806c0b6eafe04c27e40fdcf not found: ID does not exist" Nov 03 21:28:31.854656 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.854584 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f285754f-0207-4721-a4f6-62ebfe9a2223-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:31.854656 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.854621 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f285754f-0207-4721-a4f6-62ebfe9a2223-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:31.854656 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.854638 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd29t\" (UniqueName: \"kubernetes.io/projected/f285754f-0207-4721-a4f6-62ebfe9a2223-kube-api-access-xd29t\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:31.854656 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.854655 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f285754f-0207-4721-a4f6-62ebfe9a2223-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:31.949130 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.949096 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj"] Nov 03 21:28:31.952990 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:31.952963 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a77060-openshift-default-8596fdf847-xzzvj"] Nov 03 21:28:33.651491 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:33.651456 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f285754f-0207-4721-a4f6-62ebfe9a2223" path="/var/lib/kubelet/pods/f285754f-0207-4721-a4f6-62ebfe9a2223/volumes" Nov 03 21:28:35.914975 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:35.914931 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw"] Nov 03 21:28:35.915411 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:35.915199 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw" podUID="3ce14fb0-6ee8-441d-ac37-4207993f445d" containerName="maas-api" containerID="cri-o://ae3017a9e104144620da8f1097df4c7d5a309a087d0a4bac84b1d48efbb5abae" gracePeriod=30 Nov 03 21:28:36.214904 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.213937 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw" Nov 03 21:28:36.294855 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.294819 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a41883-5cbdf8bb47-m697s"] Nov 03 21:28:36.295346 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.295333 5615 cpu_manager.go:414] "containerMap: removing container" podUID="f285754f-0207-4721-a4f6-62ebfe9a2223" containerName="istio-proxy" Nov 03 21:28:36.295406 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.295348 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f285754f-0207-4721-a4f6-62ebfe9a2223" containerName="istio-proxy" Nov 03 21:28:36.295406 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.295388 5615 cpu_manager.go:414] "containerMap: removing container" podUID="f285754f-0207-4721-a4f6-62ebfe9a2223" containerName="istio-proxy" Nov 03 21:28:36.295406 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.295394 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f285754f-0207-4721-a4f6-62ebfe9a2223" containerName="istio-proxy" Nov 03 21:28:36.295500 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.295408 5615 cpu_manager.go:414] "containerMap: removing container" podUID="3ce14fb0-6ee8-441d-ac37-4207993f445d" containerName="maas-api" Nov 03 21:28:36.295500 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.295413 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ce14fb0-6ee8-441d-ac37-4207993f445d" containerName="maas-api" Nov 03 21:28:36.295500 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.295488 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="f285754f-0207-4721-a4f6-62ebfe9a2223" containerName="istio-proxy" Nov 03 21:28:36.295600 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.295503 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="f285754f-0207-4721-a4f6-62ebfe9a2223" containerName="istio-proxy" Nov 03 21:28:36.295600 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.295510 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ce14fb0-6ee8-441d-ac37-4207993f445d" containerName="maas-api" Nov 03 21:28:36.298655 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.298634 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a41883-5cbdf8bb47-m697s" Nov 03 21:28:36.306044 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.306018 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ce14fb0-6ee8-441d-ac37-4207993f445d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ce14fb0-6ee8-441d-ac37-4207993f445d-metrics-certs\") pod \"3ce14fb0-6ee8-441d-ac37-4207993f445d\" (UID: \"3ce14fb0-6ee8-441d-ac37-4207993f445d\") " Nov 03 21:28:36.306143 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.306129 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ce14fb0-6ee8-441d-ac37-4207993f445d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ce14fb0-6ee8-441d-ac37-4207993f445d-maas-api-tls\") pod \"3ce14fb0-6ee8-441d-ac37-4207993f445d\" (UID: \"3ce14fb0-6ee8-441d-ac37-4207993f445d\") " Nov 03 21:28:36.306191 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.306159 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ce14fb0-6ee8-441d-ac37-4207993f445d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ce14fb0-6ee8-441d-ac37-4207993f445d-openshift-service-ca\") pod \"3ce14fb0-6ee8-441d-ac37-4207993f445d\" (UID: \"3ce14fb0-6ee8-441d-ac37-4207993f445d\") " Nov 03 21:28:36.306229 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.306202 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ce14fb0-6ee8-441d-ac37-4207993f445d-kube-api-access-kxfxk\" (UniqueName: \"kubernetes.io/projected/3ce14fb0-6ee8-441d-ac37-4207993f445d-kube-api-access-kxfxk\") pod \"3ce14fb0-6ee8-441d-ac37-4207993f445d\" (UID: \"3ce14fb0-6ee8-441d-ac37-4207993f445d\") " Nov 03 21:28:36.306757 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.306729 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ce14fb0-6ee8-441d-ac37-4207993f445d-openshift-service-ca" pod "3ce14fb0-6ee8-441d-ac37-4207993f445d" (UID: "3ce14fb0-6ee8-441d-ac37-4207993f445d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:28:36.308600 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.308570 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ce14fb0-6ee8-441d-ac37-4207993f445d-maas-api-tls" pod "3ce14fb0-6ee8-441d-ac37-4207993f445d" (UID: "3ce14fb0-6ee8-441d-ac37-4207993f445d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:28:36.308684 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.308594 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce14fb0-6ee8-441d-ac37-4207993f445d-kube-api-access-kxfxk" pod "3ce14fb0-6ee8-441d-ac37-4207993f445d" (UID: "3ce14fb0-6ee8-441d-ac37-4207993f445d"). InnerVolumeSpecName "kube-api-access-kxfxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:28:36.308684 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.308642 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ce14fb0-6ee8-441d-ac37-4207993f445d-metrics-certs" pod "3ce14fb0-6ee8-441d-ac37-4207993f445d" (UID: "3ce14fb0-6ee8-441d-ac37-4207993f445d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:28:36.349751 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.349705 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-a41883\"" Nov 03 21:28:36.353014 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.352986 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a41883-5cbdf8bb47-m697s"] Nov 03 21:28:36.407599 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.407567 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a41883-5cbdf8bb47-m697s\" (UID: \"db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a41883-5cbdf8bb47-m697s" Nov 03 21:28:36.407599 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.407602 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtq8s\" (UniqueName: \"kubernetes.io/projected/db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5-kube-api-access-wtq8s\") pod \"payload-pre-processing-e2e-praxis-a-a41883-5cbdf8bb47-m697s\" (UID: \"db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a41883-5cbdf8bb47-m697s" Nov 03 21:28:36.407822 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.407802 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ce14fb0-6ee8-441d-ac37-4207993f445d-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:36.407877 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.407825 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ce14fb0-6ee8-441d-ac37-4207993f445d-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:36.407877 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.407837 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ce14fb0-6ee8-441d-ac37-4207993f445d-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:36.407877 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.407847 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxfxk\" (UniqueName: \"kubernetes.io/projected/3ce14fb0-6ee8-441d-ac37-4207993f445d-kube-api-access-kxfxk\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:36.411379 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.411354 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-a41883-dockercfg-gkvxr\"" Nov 03 21:28:36.441801 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.441728 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a41883-5d94544d6b-snqv5"] Nov 03 21:28:36.445785 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.445761 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a41883-5d94544d6b-snqv5" Nov 03 21:28:36.501046 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.500996 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a41883-5d94544d6b-snqv5"] Nov 03 21:28:36.511789 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.511747 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn8tv\" (UniqueName: \"kubernetes.io/projected/9df4016a-4b05-4252-8246-98e8c1e6cce4-kube-api-access-sn8tv\") pod \"payload-processing-e2e-praxis-a-a41883-5d94544d6b-snqv5\" (UID: \"9df4016a-4b05-4252-8246-98e8c1e6cce4\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a41883-5d94544d6b-snqv5" Nov 03 21:28:36.511968 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.511854 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9df4016a-4b05-4252-8246-98e8c1e6cce4-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a41883-5d94544d6b-snqv5\" (UID: \"9df4016a-4b05-4252-8246-98e8c1e6cce4\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a41883-5d94544d6b-snqv5" Nov 03 21:28:36.511968 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.511960 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a41883-5cbdf8bb47-m697s\" (UID: \"db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a41883-5cbdf8bb47-m697s" Nov 03 21:28:36.512107 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.511989 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtq8s\" (UniqueName: \"kubernetes.io/projected/db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5-kube-api-access-wtq8s\") pod \"payload-pre-processing-e2e-praxis-a-a41883-5cbdf8bb47-m697s\" (UID: \"db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a41883-5cbdf8bb47-m697s" Nov 03 21:28:36.516122 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.513249 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a41883-5cbdf8bb47-m697s\" (UID: \"db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a41883-5cbdf8bb47-m697s" Nov 03 21:28:36.576869 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.574622 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtq8s\" (UniqueName: \"kubernetes.io/projected/db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5-kube-api-access-wtq8s\") pod \"payload-pre-processing-e2e-praxis-a-a41883-5cbdf8bb47-m697s\" (UID: \"db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a41883-5cbdf8bb47-m697s" Nov 03 21:28:36.613135 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.613089 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a41883-5cbdf8bb47-m697s" Nov 03 21:28:36.613568 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.613537 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn8tv\" (UniqueName: \"kubernetes.io/projected/9df4016a-4b05-4252-8246-98e8c1e6cce4-kube-api-access-sn8tv\") pod \"payload-processing-e2e-praxis-a-a41883-5d94544d6b-snqv5\" (UID: \"9df4016a-4b05-4252-8246-98e8c1e6cce4\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a41883-5d94544d6b-snqv5" Nov 03 21:28:36.613641 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.613630 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9df4016a-4b05-4252-8246-98e8c1e6cce4-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a41883-5d94544d6b-snqv5\" (UID: \"9df4016a-4b05-4252-8246-98e8c1e6cce4\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a41883-5d94544d6b-snqv5" Nov 03 21:28:36.614419 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.614395 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9df4016a-4b05-4252-8246-98e8c1e6cce4-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a41883-5d94544d6b-snqv5\" (UID: \"9df4016a-4b05-4252-8246-98e8c1e6cce4\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a41883-5d94544d6b-snqv5" Nov 03 21:28:36.673221 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.672089 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn8tv\" (UniqueName: \"kubernetes.io/projected/9df4016a-4b05-4252-8246-98e8c1e6cce4-kube-api-access-sn8tv\") pod \"payload-processing-e2e-praxis-a-a41883-5d94544d6b-snqv5\" (UID: \"9df4016a-4b05-4252-8246-98e8c1e6cce4\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a41883-5d94544d6b-snqv5" Nov 03 21:28:36.759472 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.759434 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a41883-5d94544d6b-snqv5" Nov 03 21:28:36.812771 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.811597 5615 generic.go:356] "Generic (PLEG): container finished" podID="3ce14fb0-6ee8-441d-ac37-4207993f445d" containerID="ae3017a9e104144620da8f1097df4c7d5a309a087d0a4bac84b1d48efbb5abae" exitCode=0 Nov 03 21:28:36.812771 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.811901 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw" event={"ID":"3ce14fb0-6ee8-441d-ac37-4207993f445d","Type":"ContainerDied","Data":"ae3017a9e104144620da8f1097df4c7d5a309a087d0a4bac84b1d48efbb5abae"} Nov 03 21:28:36.812771 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.811961 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw" event={"ID":"3ce14fb0-6ee8-441d-ac37-4207993f445d","Type":"ContainerDied","Data":"d230cf3a7d6866b8e45e93e2148f249ac258a8cd3219a149f808e118b57b634f"} Nov 03 21:28:36.812771 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.811993 5615 scope.go:122] "RemoveContainer" containerID="ae3017a9e104144620da8f1097df4c7d5a309a087d0a4bac84b1d48efbb5abae" Nov 03 21:28:36.812771 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.812148 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw" Nov 03 21:28:36.834702 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.834680 5615 scope.go:122] "RemoveContainer" containerID="ae3017a9e104144620da8f1097df4c7d5a309a087d0a4bac84b1d48efbb5abae" Nov 03 21:28:36.835146 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:28:36.835123 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae3017a9e104144620da8f1097df4c7d5a309a087d0a4bac84b1d48efbb5abae\": container with ID starting with ae3017a9e104144620da8f1097df4c7d5a309a087d0a4bac84b1d48efbb5abae not found: ID does not exist" containerID="ae3017a9e104144620da8f1097df4c7d5a309a087d0a4bac84b1d48efbb5abae" Nov 03 21:28:36.835239 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:36.835154 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae3017a9e104144620da8f1097df4c7d5a309a087d0a4bac84b1d48efbb5abae"} err="failed to get container status \"ae3017a9e104144620da8f1097df4c7d5a309a087d0a4bac84b1d48efbb5abae\": rpc error: code = NotFound desc = could not find container \"ae3017a9e104144620da8f1097df4c7d5a309a087d0a4bac84b1d48efbb5abae\": container with ID starting with ae3017a9e104144620da8f1097df4c7d5a309a087d0a4bac84b1d48efbb5abae not found: ID does not exist" Nov 03 21:28:37.013168 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:37.010171 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw"] Nov 03 21:28:37.013168 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:37.010248 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b5c926-7ff7698bcc-kk2vw"] Nov 03 21:28:37.013718 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:37.013458 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a41883-5cbdf8bb47-m697s"] Nov 03 21:28:37.032681 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:28:37.032645 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb7cb3a0_c93d_4b12_9bb3_333d2c4e0ff5.slice/crio-d466826df0684711f35156c64c915af014b76f9260633401722874d3d41037aa WatchSource:0}: Error finding container d466826df0684711f35156c64c915af014b76f9260633401722874d3d41037aa: Status 404 returned error can't find the container with id d466826df0684711f35156c64c915af014b76f9260633401722874d3d41037aa Nov 03 21:28:37.137006 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:37.136979 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a41883-5d94544d6b-snqv5"] Nov 03 21:28:37.138965 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:28:37.138928 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df4016a_4b05_4252_8246_98e8c1e6cce4.slice/crio-919abad6b2a28504c1a377c61e450fdf6c61cae8a5d2c793912de00a5c3e078d WatchSource:0}: Error finding container 919abad6b2a28504c1a377c61e450fdf6c61cae8a5d2c793912de00a5c3e078d: Status 404 returned error can't find the container with id 919abad6b2a28504c1a377c61e450fdf6c61cae8a5d2c793912de00a5c3e078d Nov 03 21:28:37.651983 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:37.651950 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ce14fb0-6ee8-441d-ac37-4207993f445d" path="/var/lib/kubelet/pods/3ce14fb0-6ee8-441d-ac37-4207993f445d/volumes" Nov 03 21:28:37.819297 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:37.819257 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a41883-5d94544d6b-snqv5" event={"ID":"9df4016a-4b05-4252-8246-98e8c1e6cce4","Type":"ContainerStarted","Data":"3235f5d0965066e4eb508533d78d7ede9e6ab23e47cca352fe9fce885c746b64"} Nov 03 21:28:37.819467 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:37.819316 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a41883-5d94544d6b-snqv5" event={"ID":"9df4016a-4b05-4252-8246-98e8c1e6cce4","Type":"ContainerStarted","Data":"919abad6b2a28504c1a377c61e450fdf6c61cae8a5d2c793912de00a5c3e078d"} Nov 03 21:28:37.819467 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:37.819432 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-a41883-5d94544d6b-snqv5" Nov 03 21:28:37.820744 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:37.820716 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a41883-5cbdf8bb47-m697s" event={"ID":"db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5","Type":"ContainerStarted","Data":"4e4b3851b4a49f6b1d4f73845fa8eb366d86229ab0f14296b4f732b68bf783b4"} Nov 03 21:28:37.820882 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:37.820759 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a41883-5cbdf8bb47-m697s" event={"ID":"db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5","Type":"ContainerStarted","Data":"d466826df0684711f35156c64c915af014b76f9260633401722874d3d41037aa"} Nov 03 21:28:37.820882 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:37.820810 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a41883-5cbdf8bb47-m697s" Nov 03 21:28:37.936300 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:37.934135 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-a41883-5d94544d6b-snqv5" podStartSLOduration=1.934117606 podStartE2EDuration="1.934117606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:28:37.933486553 +0000 UTC m=+2412.401265276" watchObservedRunningTime="2026-11-03 21:28:37.934117606 +0000 UTC m=+2412.401896321" Nov 03 21:28:38.044130 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:38.044046 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a41883-5cbdf8bb47-m697s" podStartSLOduration=2.04402656 podStartE2EDuration="2.04402656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:28:38.042825687 +0000 UTC m=+2412.510604412" watchObservedRunningTime="2026-11-03 21:28:38.04402656 +0000 UTC m=+2412.511805283" Nov 03 21:28:40.620517 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:28:40.620470 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:28:40.797580 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:40.797529 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x"] Nov 03 21:28:40.797889 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:40.797790 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" podUID="09573228-1c0e-478a-b836-7b46b30e6c99" containerName="istio-proxy" containerID="cri-o://c7512935571d1586146460751075aa7feaf08f4b7f9b3176117f520491e0afd1" gracePeriod=30 Nov 03 21:28:41.821308 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:41.821263 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc"] Nov 03 21:28:41.826840 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:41.826812 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:41.880869 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:41.880824 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:41.881049 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:41.880945 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:41.881137 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:41.881045 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8gvc\" (UniqueName: \"kubernetes.io/projected/fa57678a-ec54-42d8-976c-cdaed5ee04f0-kube-api-access-x8gvc\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:41.881191 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:41.881151 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:41.881191 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:41.881188 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:41.881275 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:41.881236 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:41.881275 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:41.881271 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:41.881350 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:41.881309 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:41.881388 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:41.881349 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:41.927564 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:41.927527 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc"] Nov 03 21:28:41.982769 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:41.982735 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8gvc\" (UniqueName: \"kubernetes.io/projected/fa57678a-ec54-42d8-976c-cdaed5ee04f0-kube-api-access-x8gvc\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:41.983026 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:41.982789 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:41.983026 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:41.982818 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:41.983026 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:41.982837 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:41.983026 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:41.982853 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:41.983026 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:41.982911 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:41.983026 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:41.982947 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:41.983026 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:41.982981 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:41.983540 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:41.983102 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:41.983540 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:41.983349 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:41.983540 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:41.983409 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:41.983540 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:41.983500 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:41.983696 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:41.983589 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:41.983750 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:41.983727 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:41.985379 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:41.985355 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:41.985801 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:41.985779 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:42.042364 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:42.042324 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8gvc\" (UniqueName: \"kubernetes.io/projected/fa57678a-ec54-42d8-976c-cdaed5ee04f0-kube-api-access-x8gvc\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:42.043211 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:42.043179 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:42.144493 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:42.144396 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:42.594534 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:42.594508 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc"] Nov 03 21:28:42.597533 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:28:42.597499 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa57678a_ec54_42d8_976c_cdaed5ee04f0.slice/crio-ab509f395a8cf95a33d37b9a7e89744e0539ca3065b74422e5c24e87e29dbb02 WatchSource:0}: Error finding container ab509f395a8cf95a33d37b9a7e89744e0539ca3065b74422e5c24e87e29dbb02: Status 404 returned error can't find the container with id ab509f395a8cf95a33d37b9a7e89744e0539ca3065b74422e5c24e87e29dbb02 Nov 03 21:28:42.600514 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:42.600469 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:28:42.600598 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:42.600565 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:28:42.600644 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:42.600615 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:28:42.878224 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:42.865972 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" event={"ID":"fa57678a-ec54-42d8-976c-cdaed5ee04f0","Type":"ContainerStarted","Data":"676e2813dd1cc8e14f473ad99bc30595da9452074ef92da99b067647e7138330"} Nov 03 21:28:42.878224 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:42.866029 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" event={"ID":"fa57678a-ec54-42d8-976c-cdaed5ee04f0","Type":"ContainerStarted","Data":"ab509f395a8cf95a33d37b9a7e89744e0539ca3065b74422e5c24e87e29dbb02"} Nov 03 21:28:42.984760 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:42.984691 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" podStartSLOduration=1.9846703209999998 podStartE2EDuration="1.984670321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:28:42.981056084 +0000 UTC m=+2417.448834808" watchObservedRunningTime="2026-11-03 21:28:42.984670321 +0000 UTC m=+2417.452449045" Nov 03 21:28:43.146833 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:43.146732 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:43.148350 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:43.148317 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 21:28:43.148487 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:43.148382 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" podUID="fa57678a-ec54-42d8-976c-cdaed5ee04f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 21:28:44.146912 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:44.146870 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 21:28:44.147411 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:44.146926 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" podUID="fa57678a-ec54-42d8-976c-cdaed5ee04f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 21:28:45.147106 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:45.147038 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 21:28:45.147603 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:45.147120 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" podUID="fa57678a-ec54-42d8-976c-cdaed5ee04f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 21:28:46.133214 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.133182 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:28:46.146395 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.146358 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 21:28:46.146528 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.146413 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" podUID="fa57678a-ec54-42d8-976c-cdaed5ee04f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 21:28:46.255488 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.255446 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09573228-1c0e-478a-b836-7b46b30e6c99-kube-api-access-z2pb5\" (UniqueName: \"kubernetes.io/projected/09573228-1c0e-478a-b836-7b46b30e6c99-kube-api-access-z2pb5\") pod \"09573228-1c0e-478a-b836-7b46b30e6c99\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " Nov 03 21:28:46.255981 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.255508 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-istio-data\") pod \"09573228-1c0e-478a-b836-7b46b30e6c99\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " Nov 03 21:28:46.255981 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.255547 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09573228-1c0e-478a-b836-7b46b30e6c99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09573228-1c0e-478a-b836-7b46b30e6c99-istio-podinfo\") pod \"09573228-1c0e-478a-b836-7b46b30e6c99\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " Nov 03 21:28:46.255981 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.255573 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-workload-socket\") pod \"09573228-1c0e-478a-b836-7b46b30e6c99\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " Nov 03 21:28:46.255981 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.255651 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-credential-socket\") pod \"09573228-1c0e-478a-b836-7b46b30e6c99\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " Nov 03 21:28:46.255981 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.255748 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-workload-certs\") pod \"09573228-1c0e-478a-b836-7b46b30e6c99\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " Nov 03 21:28:46.255981 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.255812 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-istio-envoy\") pod \"09573228-1c0e-478a-b836-7b46b30e6c99\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " Nov 03 21:28:46.255981 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.255850 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-istio-data" pod "09573228-1c0e-478a-b836-7b46b30e6c99" (UID: "09573228-1c0e-478a-b836-7b46b30e6c99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:28:46.255981 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.255892 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09573228-1c0e-478a-b836-7b46b30e6c99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09573228-1c0e-478a-b836-7b46b30e6c99-istiod-ca-cert\") pod \"09573228-1c0e-478a-b836-7b46b30e6c99\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " Nov 03 21:28:46.255981 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.255879 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-workload-socket" pod "09573228-1c0e-478a-b836-7b46b30e6c99" (UID: "09573228-1c0e-478a-b836-7b46b30e6c99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:28:46.255981 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.255904 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-credential-socket" pod "09573228-1c0e-478a-b836-7b46b30e6c99" (UID: "09573228-1c0e-478a-b836-7b46b30e6c99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:28:46.255981 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.255931 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09573228-1c0e-478a-b836-7b46b30e6c99-istio-token\" (UniqueName: \"kubernetes.io/projected/09573228-1c0e-478a-b836-7b46b30e6c99-istio-token\") pod \"09573228-1c0e-478a-b836-7b46b30e6c99\" (UID: \"09573228-1c0e-478a-b836-7b46b30e6c99\") " Nov 03 21:28:46.256557 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.255988 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-workload-certs" pod "09573228-1c0e-478a-b836-7b46b30e6c99" (UID: "09573228-1c0e-478a-b836-7b46b30e6c99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:28:46.256557 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.256183 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09573228-1c0e-478a-b836-7b46b30e6c99-istiod-ca-cert" pod "09573228-1c0e-478a-b836-7b46b30e6c99" (UID: "09573228-1c0e-478a-b836-7b46b30e6c99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:28:46.256557 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.256452 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:46.256557 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.256475 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:46.256557 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.256492 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:46.256557 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.256506 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:46.256557 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.256519 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09573228-1c0e-478a-b836-7b46b30e6c99-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:46.258091 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.258052 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09573228-1c0e-478a-b836-7b46b30e6c99-istio-token" pod "09573228-1c0e-478a-b836-7b46b30e6c99" (UID: "09573228-1c0e-478a-b836-7b46b30e6c99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:28:46.258294 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.258269 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09573228-1c0e-478a-b836-7b46b30e6c99-kube-api-access-z2pb5" pod "09573228-1c0e-478a-b836-7b46b30e6c99" (UID: "09573228-1c0e-478a-b836-7b46b30e6c99"). InnerVolumeSpecName "kube-api-access-z2pb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:28:46.258294 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.258276 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09573228-1c0e-478a-b836-7b46b30e6c99-istio-podinfo" pod "09573228-1c0e-478a-b836-7b46b30e6c99" (UID: "09573228-1c0e-478a-b836-7b46b30e6c99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:28:46.258468 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.258325 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-istio-envoy" pod "09573228-1c0e-478a-b836-7b46b30e6c99" (UID: "09573228-1c0e-478a-b836-7b46b30e6c99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:28:46.357761 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.357678 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09573228-1c0e-478a-b836-7b46b30e6c99-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:46.357761 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.357711 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09573228-1c0e-478a-b836-7b46b30e6c99-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:46.357761 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.357722 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2pb5\" (UniqueName: \"kubernetes.io/projected/09573228-1c0e-478a-b836-7b46b30e6c99-kube-api-access-z2pb5\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:46.357761 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.357732 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09573228-1c0e-478a-b836-7b46b30e6c99-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:46.889841 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.889808 5615 generic.go:356] "Generic (PLEG): container finished" podID="09573228-1c0e-478a-b836-7b46b30e6c99" containerID="c7512935571d1586146460751075aa7feaf08f4b7f9b3176117f520491e0afd1" exitCode=0 Nov 03 21:28:46.890016 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.889863 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" Nov 03 21:28:46.890016 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.889890 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" event={"ID":"09573228-1c0e-478a-b836-7b46b30e6c99","Type":"ContainerDied","Data":"c7512935571d1586146460751075aa7feaf08f4b7f9b3176117f520491e0afd1"} Nov 03 21:28:46.890016 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.889946 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x" event={"ID":"09573228-1c0e-478a-b836-7b46b30e6c99","Type":"ContainerDied","Data":"3eebcc7fc1944e6a3ddd9e0a0cfa7b52203445574a47a114f19e2a61bb8a8043"} Nov 03 21:28:46.890016 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.889976 5615 scope.go:122] "RemoveContainer" containerID="c7512935571d1586146460751075aa7feaf08f4b7f9b3176117f520491e0afd1" Nov 03 21:28:46.926144 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.926115 5615 scope.go:122] "RemoveContainer" containerID="6a77bdf86d0c69ccda26c60cf9cc5dc9b8646ac2c0c2f83ef912dc04ee6cfef2" Nov 03 21:28:46.936356 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.936168 5615 scope.go:122] "RemoveContainer" containerID="c7512935571d1586146460751075aa7feaf08f4b7f9b3176117f520491e0afd1" Nov 03 21:28:46.936509 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:28:46.936489 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7512935571d1586146460751075aa7feaf08f4b7f9b3176117f520491e0afd1\": container with ID starting with c7512935571d1586146460751075aa7feaf08f4b7f9b3176117f520491e0afd1 not found: ID does not exist" containerID="c7512935571d1586146460751075aa7feaf08f4b7f9b3176117f520491e0afd1" Nov 03 21:28:46.936554 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.936517 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7512935571d1586146460751075aa7feaf08f4b7f9b3176117f520491e0afd1"} err="failed to get container status \"c7512935571d1586146460751075aa7feaf08f4b7f9b3176117f520491e0afd1\": rpc error: code = NotFound desc = could not find container \"c7512935571d1586146460751075aa7feaf08f4b7f9b3176117f520491e0afd1\": container with ID starting with c7512935571d1586146460751075aa7feaf08f4b7f9b3176117f520491e0afd1 not found: ID does not exist" Nov 03 21:28:46.936554 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.936538 5615 scope.go:122] "RemoveContainer" containerID="6a77bdf86d0c69ccda26c60cf9cc5dc9b8646ac2c0c2f83ef912dc04ee6cfef2" Nov 03 21:28:46.936795 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:28:46.936775 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a77bdf86d0c69ccda26c60cf9cc5dc9b8646ac2c0c2f83ef912dc04ee6cfef2\": container with ID starting with 6a77bdf86d0c69ccda26c60cf9cc5dc9b8646ac2c0c2f83ef912dc04ee6cfef2 not found: ID does not exist" containerID="6a77bdf86d0c69ccda26c60cf9cc5dc9b8646ac2c0c2f83ef912dc04ee6cfef2" Nov 03 21:28:46.936834 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:46.936803 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a77bdf86d0c69ccda26c60cf9cc5dc9b8646ac2c0c2f83ef912dc04ee6cfef2"} err="failed to get container status \"6a77bdf86d0c69ccda26c60cf9cc5dc9b8646ac2c0c2f83ef912dc04ee6cfef2\": rpc error: code = NotFound desc = could not find container \"6a77bdf86d0c69ccda26c60cf9cc5dc9b8646ac2c0c2f83ef912dc04ee6cfef2\": container with ID starting with 6a77bdf86d0c69ccda26c60cf9cc5dc9b8646ac2c0c2f83ef912dc04ee6cfef2 not found: ID does not exist" Nov 03 21:28:47.046639 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:47.046593 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x"] Nov 03 21:28:47.093890 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:47.093850 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-b5c926-openshift-default-77b6494b7c-qt59x"] Nov 03 21:28:47.146806 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:47.146710 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 21:28:47.146806 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:47.146774 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" podUID="fa57678a-ec54-42d8-976c-cdaed5ee04f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 21:28:47.655841 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:47.654403 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09573228-1c0e-478a-b836-7b46b30e6c99" path="/var/lib/kubelet/pods/09573228-1c0e-478a-b836-7b46b30e6c99/volumes" Nov 03 21:28:48.146436 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:48.146395 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 21:28:48.146598 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:48.146451 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" podUID="fa57678a-ec54-42d8-976c-cdaed5ee04f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 21:28:48.832060 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:48.832007 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a41883-5cbdf8bb47-m697s" Nov 03 21:28:48.832582 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:48.832271 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-a41883-5d94544d6b-snqv5" Nov 03 21:28:49.147132 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:49.147011 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 21:28:49.147132 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:49.147116 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" podUID="fa57678a-ec54-42d8-976c-cdaed5ee04f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 21:28:50.150836 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:50.150803 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:50.151372 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:50.151166 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:28:50.151806 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:28:50.151787 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:29:08.717506 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.717422 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf"] Nov 03 21:29:08.717977 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.717963 5615 cpu_manager.go:414] "containerMap: removing container" podUID="09573228-1c0e-478a-b836-7b46b30e6c99" containerName="istio-proxy" Nov 03 21:29:08.718022 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.717979 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09573228-1c0e-478a-b836-7b46b30e6c99" containerName="istio-proxy" Nov 03 21:29:08.718022 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.717988 5615 cpu_manager.go:414] "containerMap: removing container" podUID="09573228-1c0e-478a-b836-7b46b30e6c99" containerName="istio-proxy" Nov 03 21:29:08.718022 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.717994 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09573228-1c0e-478a-b836-7b46b30e6c99" containerName="istio-proxy" Nov 03 21:29:08.718157 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.718116 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="09573228-1c0e-478a-b836-7b46b30e6c99" containerName="istio-proxy" Nov 03 21:29:08.718157 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.718127 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="09573228-1c0e-478a-b836-7b46b30e6c99" containerName="istio-proxy" Nov 03 21:29:08.721873 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.721849 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:08.764438 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.764402 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27"] Nov 03 21:29:08.768318 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.768298 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:08.818335 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.818298 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf"] Nov 03 21:29:08.830873 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.830840 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b357d8d-9624-46e8-86bf-d046d06fee50-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:08.831053 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.830886 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:08.831053 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.830905 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b357d8d-9624-46e8-86bf-d046d06fee50-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:08.831053 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.831035 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:08.831242 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.831149 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:08.831242 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.831185 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b357d8d-9624-46e8-86bf-d046d06fee50-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:08.831351 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.831281 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:08.831459 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.831437 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq79k\" (UniqueName: \"kubernetes.io/projected/4b357d8d-9624-46e8-86bf-d046d06fee50-kube-api-access-nq79k\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:08.831596 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.831577 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:08.884680 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.884647 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27"] Nov 03 21:29:08.932832 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.932796 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:08.933030 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.932860 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:08.933030 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.932950 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq79k\" (UniqueName: \"kubernetes.io/projected/4b357d8d-9624-46e8-86bf-d046d06fee50-kube-api-access-nq79k\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:08.933030 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.933016 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:08.933261 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.933108 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:08.933261 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.933169 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1357b2a2-d12d-458e-942a-bdfc808972c1-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:08.933261 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.933193 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1357b2a2-d12d-458e-942a-bdfc808972c1-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:08.933261 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.933239 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:08.933489 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.933260 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:08.933489 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.933272 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1357b2a2-d12d-458e-942a-bdfc808972c1-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:08.933489 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.933329 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:08.933489 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.933356 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b357d8d-9624-46e8-86bf-d046d06fee50-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:08.933489 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.933382 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:08.933489 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.933400 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b357d8d-9624-46e8-86bf-d046d06fee50-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:08.933822 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.933492 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:08.933822 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.933533 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:08.933822 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.933535 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:08.933822 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.933634 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:08.933822 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.933670 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz469\" (UniqueName: \"kubernetes.io/projected/1357b2a2-d12d-458e-942a-bdfc808972c1-kube-api-access-bz469\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:08.933822 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.933724 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b357d8d-9624-46e8-86bf-d046d06fee50-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:08.933822 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.933806 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:08.934238 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.934069 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:08.934524 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.934500 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b357d8d-9624-46e8-86bf-d046d06fee50-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:08.936119 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.936062 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:08.936205 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.936145 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b357d8d-9624-46e8-86bf-d046d06fee50-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:08.992733 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.992691 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b357d8d-9624-46e8-86bf-d046d06fee50-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:08.992884 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:08.992815 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq79k\" (UniqueName: \"kubernetes.io/projected/4b357d8d-9624-46e8-86bf-d046d06fee50-kube-api-access-nq79k\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:09.034609 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:09.034578 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1357b2a2-d12d-458e-942a-bdfc808972c1-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:09.034609 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:09.034613 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1357b2a2-d12d-458e-942a-bdfc808972c1-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:09.034854 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:09.034633 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1357b2a2-d12d-458e-942a-bdfc808972c1-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:09.034854 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:09.034656 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:09.034854 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:09.034715 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:09.034854 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:09.034813 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz469\" (UniqueName: \"kubernetes.io/projected/1357b2a2-d12d-458e-942a-bdfc808972c1-kube-api-access-bz469\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:09.035126 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:09.034887 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:09.035126 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:09.035007 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:09.035126 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:09.035067 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:09.035307 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:09.035129 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:09.035307 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:09.035287 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:09.035416 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:09.035337 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:09.035416 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:09.035383 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1357b2a2-d12d-458e-942a-bdfc808972c1-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:09.035525 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:09.035474 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:09.035584 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:09.035532 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:09.037607 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:09.037583 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1357b2a2-d12d-458e-942a-bdfc808972c1-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:09.037607 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:09.037593 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:09.097017 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:09.096990 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1357b2a2-d12d-458e-942a-bdfc808972c1-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:09.098227 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:09.098205 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz469\" (UniqueName: \"kubernetes.io/projected/1357b2a2-d12d-458e-942a-bdfc808972c1-kube-api-access-bz469\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:09.382970 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:09.382877 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:09.458723 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:09.458695 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf"] Nov 03 21:29:09.466223 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:09.466159 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:29:09.466367 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:09.466321 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:29:09.466452 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:09.466433 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:29:09.763176 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:09.763147 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27"] Nov 03 21:29:09.766014 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:29:09.765982 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1357b2a2_d12d_458e_942a_bdfc808972c1.slice/crio-937f48c8b5378bd9db8591a10a30d8c87ee355f4bf45ee4105bccd7cee820a71 WatchSource:0}: Error finding container 937f48c8b5378bd9db8591a10a30d8c87ee355f4bf45ee4105bccd7cee820a71: Status 404 returned error can't find the container with id 937f48c8b5378bd9db8591a10a30d8c87ee355f4bf45ee4105bccd7cee820a71 Nov 03 21:29:09.768544 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:09.768505 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:29:09.768665 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:09.768590 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:29:09.768665 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:09.768639 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:29:10.030164 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:10.030030 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" event={"ID":"1357b2a2-d12d-458e-942a-bdfc808972c1","Type":"ContainerStarted","Data":"936451a7b34040edff6140da36cd5624062df0e57dfaaec1a97c74213bad5b4c"} Nov 03 21:29:10.030164 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:10.030110 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" event={"ID":"1357b2a2-d12d-458e-942a-bdfc808972c1","Type":"ContainerStarted","Data":"937f48c8b5378bd9db8591a10a30d8c87ee355f4bf45ee4105bccd7cee820a71"} Nov 03 21:29:10.031939 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:10.031908 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" event={"ID":"4b357d8d-9624-46e8-86bf-d046d06fee50","Type":"ContainerStarted","Data":"50cfe6497a22759d3f52657b444efc7b0be4e5bfc9d92ad0bf6d01dd07b72126"} Nov 03 21:29:10.032051 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:10.031961 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" event={"ID":"4b357d8d-9624-46e8-86bf-d046d06fee50","Type":"ContainerStarted","Data":"5126cad0a0cd78b124c2ca317c05c05813e6528943ac916ed01e47cc6624496e"} Nov 03 21:29:10.036311 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:10.036284 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:10.038226 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:10.038198 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 03 21:29:10.038333 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:10.038247 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" podUID="4b357d8d-9624-46e8-86bf-d046d06fee50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 03 21:29:10.147709 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:10.147654 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" podStartSLOduration=2.147637428 podStartE2EDuration="2.147637428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:10.145602757 +0000 UTC m=+2444.613381480" watchObservedRunningTime="2026-11-03 21:29:10.147637428 +0000 UTC m=+2444.615416151" Nov 03 21:29:10.262137 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:10.262060 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" podStartSLOduration=2.262046173 podStartE2EDuration="2.262046173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:10.26022225 +0000 UTC m=+2444.728000975" watchObservedRunningTime="2026-11-03 21:29:10.262046173 +0000 UTC m=+2444.729824896" Nov 03 21:29:10.384100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:10.383971 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:10.385183 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:10.385158 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 21:29:10.385245 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:10.385213 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" podUID="1357b2a2-d12d-458e-942a-bdfc808972c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 21:29:10.666946 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:29:10.666857 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:29:11.059163 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:11.056863 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 03 21:29:11.059163 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:11.056924 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" podUID="4b357d8d-9624-46e8-86bf-d046d06fee50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 03 21:29:11.383387 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:11.383298 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 21:29:11.383387 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:11.383364 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" podUID="1357b2a2-d12d-458e-942a-bdfc808972c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 21:29:12.040411 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:12.039464 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 03 21:29:12.040411 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:12.039529 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" podUID="4b357d8d-9624-46e8-86bf-d046d06fee50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 03 21:29:12.384381 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:12.384297 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 21:29:12.384381 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:12.384368 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" podUID="1357b2a2-d12d-458e-942a-bdfc808972c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 21:29:13.047100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:13.046493 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 03 21:29:13.047100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:13.046560 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" podUID="4b357d8d-9624-46e8-86bf-d046d06fee50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 03 21:29:13.383603 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:13.383524 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 21:29:13.383809 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:13.383787 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" podUID="1357b2a2-d12d-458e-942a-bdfc808972c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 21:29:14.038103 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:14.037131 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 03 21:29:14.038103 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:14.037214 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" podUID="4b357d8d-9624-46e8-86bf-d046d06fee50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 03 21:29:14.384786 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:14.384659 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 21:29:14.385098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:14.385030 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" podUID="1357b2a2-d12d-458e-942a-bdfc808972c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 21:29:15.050098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:15.042477 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 03 21:29:15.050098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:15.042543 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" podUID="4b357d8d-9624-46e8-86bf-d046d06fee50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 03 21:29:15.383451 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:15.383357 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 21:29:15.383451 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:15.383422 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" podUID="1357b2a2-d12d-458e-942a-bdfc808972c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 21:29:15.386592 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:15.386561 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7"] Nov 03 21:29:15.391772 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:15.391752 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7" Nov 03 21:29:15.451000 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:15.450948 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-a41883-serving-cert\"" Nov 03 21:29:15.492930 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:15.492893 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7"] Nov 03 21:29:15.553916 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:15.553878 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/976a1cc4-cd02-4d49-b1a1-c324af201326-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7\" (UID: \"976a1cc4-cd02-4d49-b1a1-c324af201326\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7" Nov 03 21:29:15.554262 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:15.554234 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/976a1cc4-cd02-4d49-b1a1-c324af201326-metrics-certs\") pod \"maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7\" (UID: \"976a1cc4-cd02-4d49-b1a1-c324af201326\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7" Nov 03 21:29:15.554442 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:15.554422 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcfm6\" (UniqueName: \"kubernetes.io/projected/976a1cc4-cd02-4d49-b1a1-c324af201326-kube-api-access-zcfm6\") pod \"maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7\" (UID: \"976a1cc4-cd02-4d49-b1a1-c324af201326\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7" Nov 03 21:29:15.554619 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:15.554601 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/976a1cc4-cd02-4d49-b1a1-c324af201326-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7\" (UID: \"976a1cc4-cd02-4d49-b1a1-c324af201326\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7" Nov 03 21:29:15.673787 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:15.669371 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/976a1cc4-cd02-4d49-b1a1-c324af201326-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7\" (UID: \"976a1cc4-cd02-4d49-b1a1-c324af201326\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7" Nov 03 21:29:15.680410 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:15.680362 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/976a1cc4-cd02-4d49-b1a1-c324af201326-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7\" (UID: \"976a1cc4-cd02-4d49-b1a1-c324af201326\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7" Nov 03 21:29:15.684034 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:15.680824 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/976a1cc4-cd02-4d49-b1a1-c324af201326-metrics-certs\") pod \"maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7\" (UID: \"976a1cc4-cd02-4d49-b1a1-c324af201326\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7" Nov 03 21:29:15.684897 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:15.684865 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcfm6\" (UniqueName: \"kubernetes.io/projected/976a1cc4-cd02-4d49-b1a1-c324af201326-kube-api-access-zcfm6\") pod \"maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7\" (UID: \"976a1cc4-cd02-4d49-b1a1-c324af201326\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7" Nov 03 21:29:15.695154 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:15.671624 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/976a1cc4-cd02-4d49-b1a1-c324af201326-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7\" (UID: \"976a1cc4-cd02-4d49-b1a1-c324af201326\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7" Nov 03 21:29:15.700173 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:29:15.680732 5615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-serving-cert: secret "maas-api-e2e-praxis-a-a41883-serving-cert" not found Nov 03 21:29:15.700467 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:29:15.700451 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/976a1cc4-cd02-4d49-b1a1-c324af201326-maas-api-tls podName:976a1cc4-cd02-4d49-b1a1-c324af201326 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:16.20041979 +0000 UTC m=+2450.668198496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/976a1cc4-cd02-4d49-b1a1-c324af201326-maas-api-tls") pod "maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7" (UID: "976a1cc4-cd02-4d49-b1a1-c324af201326") : secret "maas-api-e2e-praxis-a-a41883-serving-cert" not found Nov 03 21:29:15.727110 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:15.718948 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/976a1cc4-cd02-4d49-b1a1-c324af201326-metrics-certs\") pod \"maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7\" (UID: \"976a1cc4-cd02-4d49-b1a1-c324af201326\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7" Nov 03 21:29:15.750776 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:15.750738 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcfm6\" (UniqueName: \"kubernetes.io/projected/976a1cc4-cd02-4d49-b1a1-c324af201326-kube-api-access-zcfm6\") pod \"maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7\" (UID: \"976a1cc4-cd02-4d49-b1a1-c324af201326\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7" Nov 03 21:29:16.036534 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:16.036493 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 03 21:29:16.036847 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:16.036814 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" podUID="4b357d8d-9624-46e8-86bf-d046d06fee50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 03 21:29:16.299869 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:16.298868 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/976a1cc4-cd02-4d49-b1a1-c324af201326-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7\" (UID: \"976a1cc4-cd02-4d49-b1a1-c324af201326\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7" Nov 03 21:29:16.302583 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:16.302548 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/976a1cc4-cd02-4d49-b1a1-c324af201326-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7\" (UID: \"976a1cc4-cd02-4d49-b1a1-c324af201326\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7" Nov 03 21:29:16.326736 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:16.326700 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7" Nov 03 21:29:16.385307 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:16.383842 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 21:29:16.385307 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:16.383890 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" podUID="1357b2a2-d12d-458e-942a-bdfc808972c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 21:29:16.711437 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:16.711408 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7"] Nov 03 21:29:16.714799 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:29:16.714762 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod976a1cc4_cd02_4d49_b1a1_c324af201326.slice/crio-8225a65ee5c823f27962b98cddbff40c215f5cdd1377068e42ec0cd43acca5f7 WatchSource:0}: Error finding container 8225a65ee5c823f27962b98cddbff40c215f5cdd1377068e42ec0cd43acca5f7: Status 404 returned error can't find the container with id 8225a65ee5c823f27962b98cddbff40c215f5cdd1377068e42ec0cd43acca5f7 Nov 03 21:29:17.036549 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:17.036506 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 03 21:29:17.036747 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:17.036583 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" podUID="4b357d8d-9624-46e8-86bf-d046d06fee50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 03 21:29:17.177425 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:17.177385 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7" event={"ID":"976a1cc4-cd02-4d49-b1a1-c324af201326","Type":"ContainerStarted","Data":"8225a65ee5c823f27962b98cddbff40c215f5cdd1377068e42ec0cd43acca5f7"} Nov 03 21:29:18.187183 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:18.187043 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7" event={"ID":"976a1cc4-cd02-4d49-b1a1-c324af201326","Type":"ContainerStarted","Data":"41e1ca7874f6b5dba71edd638617b2508c4b5a028bf4f7dc26f0767451fcad1d"} Nov 03 21:29:18.187183 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:18.187142 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7" Nov 03 21:29:18.302552 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:18.301230 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7" podStartSLOduration=2.378806324 podStartE2EDuration="3.301211686s" totalImagesPullingTime="922.405362ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:18.29852508 +0000 UTC m=+2452.766303818" watchObservedRunningTime="2026-11-03 21:29:18.301211686 +0000 UTC m=+2452.768990411" Nov 03 21:29:18.384451 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:18.384405 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:29:18.384767 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:18.384726 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" podUID="1357b2a2-d12d-458e-942a-bdfc808972c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:29:19.076227 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:19.076187 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:29:19.076438 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:19.076247 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" podUID="4b357d8d-9624-46e8-86bf-d046d06fee50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:29:19.163302 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:19.163093 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:19.163510 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:19.163489 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:19.166659 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:19.166631 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:29:19.336771 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:19.334846 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:19.336771 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:19.336253 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:19.338044 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:19.338016 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:29:24.199483 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:24.199450 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7" Nov 03 21:29:26.813278 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:26.813248 5615 scope.go:122] "RemoveContainer" containerID="d50cce2819f0cc2a8116d79740ccd9cf8b8734cd5fc015afa8ece7d9883683ee" Nov 03 21:29:33.210525 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:33.210487 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g"] Nov 03 21:29:33.211040 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:33.210684 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g" podUID="a6cbf2de-2dc6-4747-9150-61932c13c60a" containerName="maas-api" containerID="cri-o://e6235a34b11b2f50fa1088c313a73ce8c9b7ab5420ae1c352ccffaff6198d6bf" gracePeriod=30 Nov 03 21:29:33.511593 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:33.511562 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g" Nov 03 21:29:33.640606 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:33.640556 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6cbf2de-2dc6-4747-9150-61932c13c60a-kube-api-access-mb55r\" (UniqueName: \"kubernetes.io/projected/a6cbf2de-2dc6-4747-9150-61932c13c60a-kube-api-access-mb55r\") pod \"a6cbf2de-2dc6-4747-9150-61932c13c60a\" (UID: \"a6cbf2de-2dc6-4747-9150-61932c13c60a\") " Nov 03 21:29:33.640799 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:33.640691 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6cbf2de-2dc6-4747-9150-61932c13c60a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6cbf2de-2dc6-4747-9150-61932c13c60a-openshift-service-ca\") pod \"a6cbf2de-2dc6-4747-9150-61932c13c60a\" (UID: \"a6cbf2de-2dc6-4747-9150-61932c13c60a\") " Nov 03 21:29:33.640799 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:33.640770 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6cbf2de-2dc6-4747-9150-61932c13c60a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6cbf2de-2dc6-4747-9150-61932c13c60a-maas-api-tls\") pod \"a6cbf2de-2dc6-4747-9150-61932c13c60a\" (UID: \"a6cbf2de-2dc6-4747-9150-61932c13c60a\") " Nov 03 21:29:33.640992 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:33.640858 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6cbf2de-2dc6-4747-9150-61932c13c60a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6cbf2de-2dc6-4747-9150-61932c13c60a-metrics-certs\") pod \"a6cbf2de-2dc6-4747-9150-61932c13c60a\" (UID: \"a6cbf2de-2dc6-4747-9150-61932c13c60a\") " Nov 03 21:29:33.644120 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:33.642317 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6cbf2de-2dc6-4747-9150-61932c13c60a-openshift-service-ca" pod "a6cbf2de-2dc6-4747-9150-61932c13c60a" (UID: "a6cbf2de-2dc6-4747-9150-61932c13c60a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:29:33.646531 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:33.646460 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6cbf2de-2dc6-4747-9150-61932c13c60a-metrics-certs" pod "a6cbf2de-2dc6-4747-9150-61932c13c60a" (UID: "a6cbf2de-2dc6-4747-9150-61932c13c60a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:29:33.648497 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:33.648463 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6cbf2de-2dc6-4747-9150-61932c13c60a-maas-api-tls" pod "a6cbf2de-2dc6-4747-9150-61932c13c60a" (UID: "a6cbf2de-2dc6-4747-9150-61932c13c60a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:29:33.654521 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:33.652801 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6cbf2de-2dc6-4747-9150-61932c13c60a-kube-api-access-mb55r" pod "a6cbf2de-2dc6-4747-9150-61932c13c60a" (UID: "a6cbf2de-2dc6-4747-9150-61932c13c60a"). InnerVolumeSpecName "kube-api-access-mb55r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:29:33.771796 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:33.771673 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb55r\" (UniqueName: \"kubernetes.io/projected/a6cbf2de-2dc6-4747-9150-61932c13c60a-kube-api-access-mb55r\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:33.771796 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:33.771725 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6cbf2de-2dc6-4747-9150-61932c13c60a-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:33.771796 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:33.771745 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6cbf2de-2dc6-4747-9150-61932c13c60a-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:33.771796 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:33.771762 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6cbf2de-2dc6-4747-9150-61932c13c60a-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:34.290362 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:34.290324 5615 generic.go:356] "Generic (PLEG): container finished" podID="a6cbf2de-2dc6-4747-9150-61932c13c60a" containerID="e6235a34b11b2f50fa1088c313a73ce8c9b7ab5420ae1c352ccffaff6198d6bf" exitCode=0 Nov 03 21:29:34.290784 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:34.290372 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g" Nov 03 21:29:34.290784 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:34.290407 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g" event={"ID":"a6cbf2de-2dc6-4747-9150-61932c13c60a","Type":"ContainerDied","Data":"e6235a34b11b2f50fa1088c313a73ce8c9b7ab5420ae1c352ccffaff6198d6bf"} Nov 03 21:29:34.290784 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:34.290461 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g" event={"ID":"a6cbf2de-2dc6-4747-9150-61932c13c60a","Type":"ContainerDied","Data":"e842487aafc7f13ba52f299e0acaf6e666d2671eaaa9cb204cf005b50d0e7c53"} Nov 03 21:29:34.290784 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:34.290485 5615 scope.go:122] "RemoveContainer" containerID="e6235a34b11b2f50fa1088c313a73ce8c9b7ab5420ae1c352ccffaff6198d6bf" Nov 03 21:29:34.300045 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:34.300013 5615 scope.go:122] "RemoveContainer" containerID="e6235a34b11b2f50fa1088c313a73ce8c9b7ab5420ae1c352ccffaff6198d6bf" Nov 03 21:29:34.300396 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:29:34.300372 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6235a34b11b2f50fa1088c313a73ce8c9b7ab5420ae1c352ccffaff6198d6bf\": container with ID starting with e6235a34b11b2f50fa1088c313a73ce8c9b7ab5420ae1c352ccffaff6198d6bf not found: ID does not exist" containerID="e6235a34b11b2f50fa1088c313a73ce8c9b7ab5420ae1c352ccffaff6198d6bf" Nov 03 21:29:34.300451 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:34.300409 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6235a34b11b2f50fa1088c313a73ce8c9b7ab5420ae1c352ccffaff6198d6bf"} err="failed to get container status \"e6235a34b11b2f50fa1088c313a73ce8c9b7ab5420ae1c352ccffaff6198d6bf\": rpc error: code = NotFound desc = could not find container \"e6235a34b11b2f50fa1088c313a73ce8c9b7ab5420ae1c352ccffaff6198d6bf\": container with ID starting with e6235a34b11b2f50fa1088c313a73ce8c9b7ab5420ae1c352ccffaff6198d6bf not found: ID does not exist" Nov 03 21:29:34.447599 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:34.447557 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g"] Nov 03 21:29:34.494795 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:34.494762 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2285b9-54bd5f89d5-8wb7g"] Nov 03 21:29:34.908391 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:34.908354 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx"] Nov 03 21:29:34.908925 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:34.908911 5615 cpu_manager.go:414] "containerMap: removing container" podUID="a6cbf2de-2dc6-4747-9150-61932c13c60a" containerName="maas-api" Nov 03 21:29:34.908979 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:34.908926 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6cbf2de-2dc6-4747-9150-61932c13c60a" containerName="maas-api" Nov 03 21:29:34.909066 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:34.909057 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6cbf2de-2dc6-4747-9150-61932c13c60a" containerName="maas-api" Nov 03 21:29:34.914017 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:34.913989 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:34.963903 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:34.963872 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-c70fca-openshift-default-dockercfg-qs65x\"" Nov 03 21:29:34.971557 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:34.971525 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx"] Nov 03 21:29:34.995618 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:34.995583 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istio-token\") pod \"e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:34.995832 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:34.995627 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istiod-ca-cert\") pod \"e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:34.995832 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:34.995740 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-workload-socket\") pod \"e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:34.995832 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:34.995780 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-credential-socket\") pod \"e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:34.996026 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:34.995861 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istio-podinfo\") pod \"e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:34.996026 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:34.995911 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istio-envoy\") pod \"e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:34.996026 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:34.995945 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istio-data\") pod \"e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:34.996026 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:34.995967 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-workload-certs\") pod \"e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:34.996272 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:34.996066 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgxsh\" (UniqueName: \"kubernetes.io/projected/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-kube-api-access-kgxsh\") pod \"e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:35.105097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:35.103871 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgxsh\" (UniqueName: \"kubernetes.io/projected/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-kube-api-access-kgxsh\") pod \"e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:35.105097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:35.104014 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istio-token\") pod \"e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:35.105097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:35.104060 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istiod-ca-cert\") pod \"e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:35.105097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:35.104131 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-workload-socket\") pod \"e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:35.105097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:35.104158 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-credential-socket\") pod \"e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:35.105097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:35.104197 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istio-podinfo\") pod \"e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:35.105097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:35.104267 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istio-envoy\") pod \"e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:35.105097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:35.104321 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istio-data\") pod \"e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:35.105097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:35.104355 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-workload-certs\") pod \"e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:35.105097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:35.104930 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-workload-certs\") pod \"e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:35.109099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:35.106515 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-credential-socket\") pod \"e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:35.109099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:35.107199 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istiod-ca-cert\") pod \"e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:35.109099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:35.108196 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istio-data\") pod \"e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:35.124098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:35.107479 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-workload-socket\") pod \"e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:35.152849 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:35.151989 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istio-podinfo\") pod \"e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:35.169634 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:35.169546 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istio-envoy\") pod \"e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:35.181096 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:35.176723 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgxsh\" (UniqueName: \"kubernetes.io/projected/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-kube-api-access-kgxsh\") pod \"e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:35.181096 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:35.178198 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istio-token\") pod \"e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:35.228181 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:35.228146 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:35.661671 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:35.661636 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6cbf2de-2dc6-4747-9150-61932c13c60a" path="/var/lib/kubelet/pods/a6cbf2de-2dc6-4747-9150-61932c13c60a/volumes" Nov 03 21:29:35.673654 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:35.673624 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx"] Nov 03 21:29:35.675514 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:29:35.675486 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod782d9d6a_583d_4bdc_af1e_d87d81db3d4a.slice/crio-d60d45f79953c6eec7ee0e3f77d91aed1ba8756d1d4b469f721d1bac15aadf35 WatchSource:0}: Error finding container d60d45f79953c6eec7ee0e3f77d91aed1ba8756d1d4b469f721d1bac15aadf35: Status 404 returned error can't find the container with id d60d45f79953c6eec7ee0e3f77d91aed1ba8756d1d4b469f721d1bac15aadf35 Nov 03 21:29:35.677896 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:35.677861 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:29:35.677977 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:35.677928 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:29:35.677977 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:35.677957 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:29:36.306963 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:36.306925 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" event={"ID":"782d9d6a-583d-4bdc-af1e-d87d81db3d4a","Type":"ContainerStarted","Data":"aa5d5e70a16b8e2a7c9bc5b90e58a014962e173d9931e48f17f47cd709d08ca2"} Nov 03 21:29:36.307218 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:36.306982 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" event={"ID":"782d9d6a-583d-4bdc-af1e-d87d81db3d4a","Type":"ContainerStarted","Data":"d60d45f79953c6eec7ee0e3f77d91aed1ba8756d1d4b469f721d1bac15aadf35"} Nov 03 21:29:36.422546 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:36.422488 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" podStartSLOduration=2.422472863 podStartE2EDuration="2.422472863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:36.421657243 +0000 UTC m=+2470.889435967" watchObservedRunningTime="2026-11-03 21:29:36.422472863 +0000 UTC m=+2470.890251585" Nov 03 21:29:37.229124 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:37.229062 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:37.235044 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:37.235011 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:37.313975 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:37.313943 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:37.315134 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:37.315111 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:29:40.847104 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:29:40.840283 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:29:41.285136 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.285057 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5"] Nov 03 21:29:41.289446 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.289419 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:41.339198 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.339155 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-9818d8-openshift-default-dockercfg-x2t57\"" Nov 03 21:29:41.359841 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.359800 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-istio-data\") pod \"e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:41.360051 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.359868 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-workload-certs\") pod \"e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:41.360051 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.359961 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88203e05-6890-4c96-9d15-07c6685ad6d3-istio-token\") pod \"e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:41.360051 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.360002 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88203e05-6890-4c96-9d15-07c6685ad6d3-istiod-ca-cert\") pod \"e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:41.360051 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.360028 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-workload-socket\") pod \"e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:41.360260 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.360111 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88203e05-6890-4c96-9d15-07c6685ad6d3-istio-podinfo\") pod \"e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:41.360260 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.360145 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-istio-envoy\") pod \"e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:41.360260 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.360185 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvzsk\" (UniqueName: \"kubernetes.io/projected/88203e05-6890-4c96-9d15-07c6685ad6d3-kube-api-access-wvzsk\") pod \"e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:41.360260 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.360221 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-credential-socket\") pod \"e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:41.390244 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.390211 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5"] Nov 03 21:29:41.472306 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.472262 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-istio-data\") pod \"e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:41.472626 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.472603 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-workload-certs\") pod \"e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:41.472762 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.472746 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88203e05-6890-4c96-9d15-07c6685ad6d3-istio-token\") pod \"e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:41.472879 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.472865 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88203e05-6890-4c96-9d15-07c6685ad6d3-istiod-ca-cert\") pod \"e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:41.472999 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.472984 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-workload-socket\") pod \"e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:41.473190 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.473171 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88203e05-6890-4c96-9d15-07c6685ad6d3-istio-podinfo\") pod \"e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:41.473338 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.473320 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-istio-envoy\") pod \"e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:41.473501 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.473481 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvzsk\" (UniqueName: \"kubernetes.io/projected/88203e05-6890-4c96-9d15-07c6685ad6d3-kube-api-access-wvzsk\") pod \"e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:41.473663 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.473647 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-credential-socket\") pod \"e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:41.474343 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.474320 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-credential-socket\") pod \"e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:41.478769 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.474720 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-workload-socket\") pod \"e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:41.478769 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.478687 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88203e05-6890-4c96-9d15-07c6685ad6d3-istiod-ca-cert\") pod \"e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:41.479104 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.479057 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-istio-data\") pod \"e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:41.479377 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.479361 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-workload-certs\") pod \"e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:41.485256 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.484883 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-istio-envoy\") pod \"e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:41.487014 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.486989 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88203e05-6890-4c96-9d15-07c6685ad6d3-istio-podinfo\") pod \"e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:41.538029 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.537942 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88203e05-6890-4c96-9d15-07c6685ad6d3-istio-token\") pod \"e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:41.543126 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.543094 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvzsk\" (UniqueName: \"kubernetes.io/projected/88203e05-6890-4c96-9d15-07c6685ad6d3-kube-api-access-wvzsk\") pod \"e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:41.624788 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:41.624748 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:42.153799 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:42.153769 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5"] Nov 03 21:29:42.155898 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:29:42.155865 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88203e05_6890_4c96_9d15_07c6685ad6d3.slice/crio-c815e0a612e949d0bed7f36495ae501502d8c7868b6529ea2a4efddf2a3df4cd WatchSource:0}: Error finding container c815e0a612e949d0bed7f36495ae501502d8c7868b6529ea2a4efddf2a3df4cd: Status 404 returned error can't find the container with id c815e0a612e949d0bed7f36495ae501502d8c7868b6529ea2a4efddf2a3df4cd Nov 03 21:29:42.158472 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:42.158438 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:29:42.158544 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:42.158511 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:29:42.158586 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:42.158542 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:29:42.528372 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:42.528331 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" event={"ID":"88203e05-6890-4c96-9d15-07c6685ad6d3","Type":"ContainerStarted","Data":"1b9eab921b1dfd61588c0517bc335ac9509661386f69c4ff6c9398c4f29cc7ca"} Nov 03 21:29:42.528530 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:42.528386 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" event={"ID":"88203e05-6890-4c96-9d15-07c6685ad6d3","Type":"ContainerStarted","Data":"c815e0a612e949d0bed7f36495ae501502d8c7868b6529ea2a4efddf2a3df4cd"} Nov 03 21:29:42.625401 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:42.625356 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:42.643949 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:42.643890 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" podStartSLOduration=1.643874965 podStartE2EDuration="1.643874965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:42.643653805 +0000 UTC m=+2477.111432552" watchObservedRunningTime="2026-11-03 21:29:42.643874965 +0000 UTC m=+2477.111653685" Nov 03 21:29:43.631161 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:43.631129 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:44.540711 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:44.540664 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:44.541779 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:44.541757 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:29:49.594368 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:49.594330 5615 generic.go:356] "Generic (PLEG): container finished" podID="4c842d3a-c0e4-4703-acc7-77053dd626ba" containerID="b6a0b200e52325e74816d5357db26ea82cfb519572aedac14ef6e19ea27b4e3d" exitCode=0 Nov 03 21:29:49.594869 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:49.594396 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" event={"ID":"4c842d3a-c0e4-4703-acc7-77053dd626ba","Type":"ContainerDied","Data":"b6a0b200e52325e74816d5357db26ea82cfb519572aedac14ef6e19ea27b4e3d"} Nov 03 21:29:49.594869 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:49.594801 5615 scope.go:122] "RemoveContainer" containerID="b6a0b200e52325e74816d5357db26ea82cfb519572aedac14ef6e19ea27b4e3d" Nov 03 21:29:49.596408 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:49.596376 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:29:49.596504 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:49.596465 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:29:49.596504 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:49.596499 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:29:50.631388 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:50.631099 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" event={"ID":"4c842d3a-c0e4-4703-acc7-77053dd626ba","Type":"ContainerStarted","Data":"656a2c2d142b7a5add4a0da2b96aa2decb24e56fbece2759e5c831699cb58cd5"} Nov 03 21:29:51.146896 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:51.146854 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:29:51.147122 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:51.147069 5615 patch_prober.go:29] interesting pod/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 03 21:29:51.147177 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:51.147149 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" podUID="4c842d3a-c0e4-4703-acc7-77053dd626ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 03 21:29:52.146563 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:52.146528 5615 patch_prober.go:29] interesting pod/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 03 21:29:52.147120 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:52.146589 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" podUID="4c842d3a-c0e4-4703-acc7-77053dd626ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 03 21:29:53.146689 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:53.146648 5615 patch_prober.go:29] interesting pod/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 03 21:29:53.147151 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:53.146726 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" podUID="4c842d3a-c0e4-4703-acc7-77053dd626ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 03 21:29:54.146767 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:54.146727 5615 patch_prober.go:29] interesting pod/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 03 21:29:54.147269 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:54.146785 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" podUID="4c842d3a-c0e4-4703-acc7-77053dd626ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 03 21:29:55.146835 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:55.146797 5615 patch_prober.go:29] interesting pod/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 03 21:29:55.147337 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:55.146872 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" podUID="4c842d3a-c0e4-4703-acc7-77053dd626ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 03 21:29:56.150969 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:56.150937 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:29:56.151478 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:56.151414 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:29:56.152007 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:56.151986 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:29:57.465562 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.465495 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc"] Nov 03 21:29:57.475765 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.475731 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:29:57.523503 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.523468 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s"] Nov 03 21:29:57.525739 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.525655 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:29:57.525943 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.525836 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:29:57.525943 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.525895 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:29:57.525943 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.525930 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:29:57.526173 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.526048 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:29:57.526173 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.526148 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:29:57.526277 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.526182 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:29:57.526362 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.526342 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfw49\" (UniqueName: \"kubernetes.io/projected/7242ebfd-977b-4b38-9c9f-f9c125ef3280-kube-api-access-gfw49\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:29:57.526422 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.526410 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:29:57.527800 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.527785 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s" Nov 03 21:29:57.577446 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.577413 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-9818d8\"" Nov 03 21:29:57.577619 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.577543 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-9818d8-dockercfg-bb9lm\"" Nov 03 21:29:57.592305 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.592272 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc"] Nov 03 21:29:57.597899 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.597859 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s"] Nov 03 21:29:57.686957 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.686910 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:29:57.691287 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.687408 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfw49\" (UniqueName: \"kubernetes.io/projected/7242ebfd-977b-4b38-9c9f-f9c125ef3280-kube-api-access-gfw49\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:29:57.691593 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.691563 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8v58\" (UniqueName: \"kubernetes.io/projected/cb894762-26e0-4dd6-9e55-0a0aed4455c9-kube-api-access-p8v58\") pod \"payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s\" (UID: \"cb894762-26e0-4dd6-9e55-0a0aed4455c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s" Nov 03 21:29:57.691778 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.691760 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb894762-26e0-4dd6-9e55-0a0aed4455c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s\" (UID: \"cb894762-26e0-4dd6-9e55-0a0aed4455c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s" Nov 03 21:29:57.691903 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.691886 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:29:57.692036 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.692020 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:29:57.692267 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.692251 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:29:57.692397 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.692384 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:29:57.692506 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.692491 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:29:57.692660 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.692646 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:29:57.692791 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.692778 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:29:57.693605 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.693573 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:29:57.719850 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.699561 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:29:57.719850 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.699815 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:29:57.719850 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.708700 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:29:57.719850 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.709104 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:29:57.728537 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.728458 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:29:57.742093 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.742042 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:29:57.753739 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.753706 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:29:57.754283 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.754257 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfw49\" (UniqueName: \"kubernetes.io/projected/7242ebfd-977b-4b38-9c9f-f9c125ef3280-kube-api-access-gfw49\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:29:57.794616 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.794576 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb894762-26e0-4dd6-9e55-0a0aed4455c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s\" (UID: \"cb894762-26e0-4dd6-9e55-0a0aed4455c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s" Nov 03 21:29:57.794880 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.794857 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8v58\" (UniqueName: \"kubernetes.io/projected/cb894762-26e0-4dd6-9e55-0a0aed4455c9-kube-api-access-p8v58\") pod \"payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s\" (UID: \"cb894762-26e0-4dd6-9e55-0a0aed4455c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s" Nov 03 21:29:57.795440 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.795414 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb894762-26e0-4dd6-9e55-0a0aed4455c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s\" (UID: \"cb894762-26e0-4dd6-9e55-0a0aed4455c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s" Nov 03 21:29:57.800695 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.800660 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:29:57.854018 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:57.853985 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8v58\" (UniqueName: \"kubernetes.io/projected/cb894762-26e0-4dd6-9e55-0a0aed4455c9-kube-api-access-p8v58\") pod \"payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s\" (UID: \"cb894762-26e0-4dd6-9e55-0a0aed4455c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s" Nov 03 21:29:58.141867 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:58.141831 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s" Nov 03 21:29:58.189817 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:58.189782 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc"] Nov 03 21:29:58.191699 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:29:58.191659 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7242ebfd_977b_4b38_9c9f_f9c125ef3280.slice/crio-71fb4fabb2c3244f22f9fbdca1bf5a26500216ce0f145fe47a705623948c4460 WatchSource:0}: Error finding container 71fb4fabb2c3244f22f9fbdca1bf5a26500216ce0f145fe47a705623948c4460: Status 404 returned error can't find the container with id 71fb4fabb2c3244f22f9fbdca1bf5a26500216ce0f145fe47a705623948c4460 Nov 03 21:29:58.194952 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:58.194913 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:29:58.195110 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:58.195002 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:29:58.195110 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:58.195042 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:29:58.538099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:58.538008 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s"] Nov 03 21:29:58.560245 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:29:58.560204 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb894762_26e0_4dd6_9e55_0a0aed4455c9.slice/crio-a40a6fd3ddb1e559f816f90dfbbddbd6aec5108fdba741fb596d9d782a991a81 WatchSource:0}: Error finding container a40a6fd3ddb1e559f816f90dfbbddbd6aec5108fdba741fb596d9d782a991a81: Status 404 returned error can't find the container with id a40a6fd3ddb1e559f816f90dfbbddbd6aec5108fdba741fb596d9d782a991a81 Nov 03 21:29:58.762919 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:58.758905 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" event={"ID":"7242ebfd-977b-4b38-9c9f-f9c125ef3280","Type":"ContainerStarted","Data":"ee5e48254a7196b196918bd8f446da4000f995ac12feadbb3630173d0e3a1c6b"} Nov 03 21:29:58.762919 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:58.758973 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" event={"ID":"7242ebfd-977b-4b38-9c9f-f9c125ef3280","Type":"ContainerStarted","Data":"71fb4fabb2c3244f22f9fbdca1bf5a26500216ce0f145fe47a705623948c4460"} Nov 03 21:29:58.762919 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:58.762435 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s" event={"ID":"cb894762-26e0-4dd6-9e55-0a0aed4455c9","Type":"ContainerStarted","Data":"a40a6fd3ddb1e559f816f90dfbbddbd6aec5108fdba741fb596d9d782a991a81"} Nov 03 21:29:58.801392 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:58.801246 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:29:59.829008 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:59.828971 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 21:29:59.830059 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:59.830018 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" podUID="7242ebfd-977b-4b38-9c9f-f9c125ef3280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 21:29:59.867102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:59.838027 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s" event={"ID":"cb894762-26e0-4dd6-9e55-0a0aed4455c9","Type":"ContainerStarted","Data":"426167e2eb127abef614e6002ae6b0e53dd17cfd61b7e005ca240e1071d557e0"} Nov 03 21:29:59.868488 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:59.868448 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s" Nov 03 21:29:59.969096 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:29:59.965815 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" podStartSLOduration=2.965795817 podStartE2EDuration="2.965795817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:58.879410431 +0000 UTC m=+2493.347189156" watchObservedRunningTime="2026-11-03 21:29:59.965795817 +0000 UTC m=+2494.433574541" Nov 03 21:30:00.020152 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:00.006706 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s" podStartSLOduration=3.006691666 podStartE2EDuration="3.006691666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:59.962461083 +0000 UTC m=+2494.430239808" watchObservedRunningTime="2026-11-03 21:30:00.006691666 +0000 UTC m=+2494.474470394" Nov 03 21:30:00.810108 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:00.801539 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 21:30:00.810455 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:00.810412 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" podUID="7242ebfd-977b-4b38-9c9f-f9c125ef3280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 21:30:01.808569 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:01.808525 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 21:30:01.809219 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:01.809185 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" podUID="7242ebfd-977b-4b38-9c9f-f9c125ef3280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 21:30:02.440743 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.440709 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k"] Nov 03 21:30:02.444372 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.444348 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:02.494858 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.494818 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-c11538-openshift-default-dockercfg-sjr44\"" Nov 03 21:30:02.532792 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.532730 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77818da7-0c48-434f-b2cb-68d47b3c2a33-istio-token\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:02.532968 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.532810 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-istio-envoy\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:02.533054 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.532958 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-workload-certs\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:02.533054 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.533017 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-credential-socket\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:02.533054 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.533045 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-workload-socket\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:02.533237 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.533123 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77818da7-0c48-434f-b2cb-68d47b3c2a33-istiod-ca-cert\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:02.533237 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.533165 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv29b\" (UniqueName: \"kubernetes.io/projected/77818da7-0c48-434f-b2cb-68d47b3c2a33-kube-api-access-nv29b\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:02.533329 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.533202 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77818da7-0c48-434f-b2cb-68d47b3c2a33-istio-podinfo\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:02.533329 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.533296 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-istio-data\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:02.544156 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.544121 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k"] Nov 03 21:30:02.637099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.634930 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-workload-certs\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:02.637099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.635000 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-credential-socket\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:02.637099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.635031 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-workload-socket\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:02.637099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.635116 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77818da7-0c48-434f-b2cb-68d47b3c2a33-istiod-ca-cert\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:02.637099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.635157 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv29b\" (UniqueName: \"kubernetes.io/projected/77818da7-0c48-434f-b2cb-68d47b3c2a33-kube-api-access-nv29b\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:02.637099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.635195 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77818da7-0c48-434f-b2cb-68d47b3c2a33-istio-podinfo\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:02.637099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.635246 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-istio-data\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:02.637099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.635493 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77818da7-0c48-434f-b2cb-68d47b3c2a33-istio-token\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:02.637099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.635545 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-istio-envoy\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:02.653100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.637871 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77818da7-0c48-434f-b2cb-68d47b3c2a33-istiod-ca-cert\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:02.653100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.638168 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-workload-certs\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:02.653100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.638525 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-credential-socket\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:02.653100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.638771 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-workload-socket\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:02.653100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.639007 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-istio-data\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:02.696200 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.685197 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-istio-envoy\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:02.696200 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.685755 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77818da7-0c48-434f-b2cb-68d47b3c2a33-istio-podinfo\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:02.772667 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.772619 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv29b\" (UniqueName: \"kubernetes.io/projected/77818da7-0c48-434f-b2cb-68d47b3c2a33-kube-api-access-nv29b\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:02.773773 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.773719 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77818da7-0c48-434f-b2cb-68d47b3c2a33-istio-token\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:02.818133 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.804475 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 21:30:02.818133 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:02.804547 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" podUID="7242ebfd-977b-4b38-9c9f-f9c125ef3280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 21:30:03.058324 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:03.058277 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:03.626165 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:03.626122 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k"] Nov 03 21:30:03.648885 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:30:03.648855 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77818da7_0c48_434f_b2cb_68d47b3c2a33.slice/crio-c484cf5692862b0208164eec64362a8eee9d82fd3463780b9fc31a17f886f93f WatchSource:0}: Error finding container c484cf5692862b0208164eec64362a8eee9d82fd3463780b9fc31a17f886f93f: Status 404 returned error can't find the container with id c484cf5692862b0208164eec64362a8eee9d82fd3463780b9fc31a17f886f93f Nov 03 21:30:03.652113 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:03.651814 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:30:03.652113 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:03.651929 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:30:03.652113 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:03.651972 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:30:03.805905 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:03.805870 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 21:30:03.806190 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:03.806161 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" podUID="7242ebfd-977b-4b38-9c9f-f9c125ef3280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 21:30:03.894764 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:03.894680 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" event={"ID":"77818da7-0c48-434f-b2cb-68d47b3c2a33","Type":"ContainerStarted","Data":"be7edeb286d1dda785653b4dcfada1800ead4f1fc7961349c0f7246bd05b8f58"} Nov 03 21:30:03.894764 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:03.894736 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" event={"ID":"77818da7-0c48-434f-b2cb-68d47b3c2a33","Type":"ContainerStarted","Data":"c484cf5692862b0208164eec64362a8eee9d82fd3463780b9fc31a17f886f93f"} Nov 03 21:30:04.058532 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:04.058499 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:04.059751 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:04.059720 5615 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 21:30:04.059883 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:04.059776 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" podUID="77818da7-0c48-434f-b2cb-68d47b3c2a33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 21:30:04.815100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:04.809513 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 21:30:04.815100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:04.809588 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" podUID="7242ebfd-977b-4b38-9c9f-f9c125ef3280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 21:30:05.063637 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:05.063604 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:05.180636 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:05.180523 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" podStartSLOduration=3.180508378 podStartE2EDuration="3.180508378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:04.043035805 +0000 UTC m=+2498.510814541" watchObservedRunningTime="2026-11-03 21:30:05.180508378 +0000 UTC m=+2499.648287100" Nov 03 21:30:05.910692 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:05.910655 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:05.911673 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:05.911644 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:30:06.262449 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.262383 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp"] Nov 03 21:30:06.270598 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.270561 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:06.395304 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.395261 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp"] Nov 03 21:30:06.396601 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.396561 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-cc4de5-openshift-default-dockercfg-gqpnq\"" Nov 03 21:30:06.399613 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.399579 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-istio-envoy\") pod \"e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:06.399753 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.399639 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618668eb-db73-4af8-967f-f73331c493f2-istio-podinfo\") pod \"e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:06.399753 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.399667 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618668eb-db73-4af8-967f-f73331c493f2-istio-token\") pod \"e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:06.399753 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.399716 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-workload-certs\") pod \"e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:06.399753 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.399747 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29m4x\" (UniqueName: \"kubernetes.io/projected/618668eb-db73-4af8-967f-f73331c493f2-kube-api-access-29m4x\") pod \"e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:06.399954 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.399797 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618668eb-db73-4af8-967f-f73331c493f2-istiod-ca-cert\") pod \"e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:06.399954 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.399852 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-istio-data\") pod \"e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:06.399954 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.399912 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-workload-socket\") pod \"e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:06.400158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.400045 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-credential-socket\") pod \"e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:06.513053 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.511398 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-workload-socket\") pod \"e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:06.513053 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.511558 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-credential-socket\") pod \"e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:06.513053 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.511638 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-istio-envoy\") pod \"e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:06.513053 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.511676 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618668eb-db73-4af8-967f-f73331c493f2-istio-podinfo\") pod \"e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:06.513053 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.511701 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618668eb-db73-4af8-967f-f73331c493f2-istio-token\") pod \"e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:06.513053 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.511755 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-workload-certs\") pod \"e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:06.513053 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.511787 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29m4x\" (UniqueName: \"kubernetes.io/projected/618668eb-db73-4af8-967f-f73331c493f2-kube-api-access-29m4x\") pod \"e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:06.513053 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.511844 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618668eb-db73-4af8-967f-f73331c493f2-istiod-ca-cert\") pod \"e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:06.513053 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.511900 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-istio-data\") pod \"e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:06.534176 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.534133 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-workload-certs\") pod \"e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:06.534468 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.534435 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-istio-data\") pod \"e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:06.534656 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.534233 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-credential-socket\") pod \"e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:06.534789 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.534163 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-workload-socket\") pod \"e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:06.535912 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.535883 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618668eb-db73-4af8-967f-f73331c493f2-istiod-ca-cert\") pod \"e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:06.538290 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.538262 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618668eb-db73-4af8-967f-f73331c493f2-istio-podinfo\") pod \"e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:06.539398 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.539313 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-istio-envoy\") pod \"e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:06.573049 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.573018 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29m4x\" (UniqueName: \"kubernetes.io/projected/618668eb-db73-4af8-967f-f73331c493f2-kube-api-access-29m4x\") pod \"e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:06.579475 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.579446 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618668eb-db73-4af8-967f-f73331c493f2-istio-token\") pod \"e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:06.585542 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.585511 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:06.803593 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.802648 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:30:06.803593 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:06.802712 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" podUID="7242ebfd-977b-4b38-9c9f-f9c125ef3280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:30:07.030799 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:07.030759 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp"] Nov 03 21:30:07.033834 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:30:07.033801 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod618668eb_db73_4af8_967f_f73331c493f2.slice/crio-70ea49a92b87c07828fdbc1cdad0dcd7b27abf7cb18145b41554dc74c548650a WatchSource:0}: Error finding container 70ea49a92b87c07828fdbc1cdad0dcd7b27abf7cb18145b41554dc74c548650a: Status 404 returned error can't find the container with id 70ea49a92b87c07828fdbc1cdad0dcd7b27abf7cb18145b41554dc74c548650a Nov 03 21:30:07.036760 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:07.036724 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:30:07.036874 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:07.036807 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:30:07.036874 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:07.036855 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:30:07.669339 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:07.669297 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:30:07.670824 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:07.670796 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:30:07.671551 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:07.671530 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:30:07.928224 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:07.928018 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" event={"ID":"618668eb-db73-4af8-967f-f73331c493f2","Type":"ContainerStarted","Data":"d18982066814a79e6da0455fccfcf4a09093d54b440c74083b43531b6929c5ed"} Nov 03 21:30:07.928224 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:07.928204 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" event={"ID":"618668eb-db73-4af8-967f-f73331c493f2","Type":"ContainerStarted","Data":"70ea49a92b87c07828fdbc1cdad0dcd7b27abf7cb18145b41554dc74c548650a"} Nov 03 21:30:08.048492 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:08.048435 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" podStartSLOduration=2.048419753 podStartE2EDuration="2.048419753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:08.045690106 +0000 UTC m=+2502.513468830" watchObservedRunningTime="2026-11-03 21:30:08.048419753 +0000 UTC m=+2502.516198475" Nov 03 21:30:08.586351 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:08.586312 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:08.591783 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:08.591756 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:08.945226 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:08.945104 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:08.946146 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:08.946122 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:30:10.619567 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:30:10.619527 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:30:11.881742 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:11.881712 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s" Nov 03 21:30:15.204105 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:15.203953 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-c11538-d9df47586-gj7tm"] Nov 03 21:30:15.234100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:15.230368 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c11538-d9df47586-gj7tm" Nov 03 21:30:15.243098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:15.236154 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-c11538-d9df47586-gj7tm"] Nov 03 21:30:15.287457 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:15.287420 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-c11538\"" Nov 03 21:30:15.288831 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:15.288794 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c90b432-31d5-49c0-8f4e-11397476f868-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-c11538-d9df47586-gj7tm\" (UID: \"5c90b432-31d5-49c0-8f4e-11397476f868\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c11538-d9df47586-gj7tm" Nov 03 21:30:15.288966 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:15.288850 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdfkf\" (UniqueName: \"kubernetes.io/projected/5c90b432-31d5-49c0-8f4e-11397476f868-kube-api-access-kdfkf\") pod \"payload-processing-e2e-shared-a-w0-c11538-d9df47586-gj7tm\" (UID: \"5c90b432-31d5-49c0-8f4e-11397476f868\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c11538-d9df47586-gj7tm" Nov 03 21:30:15.334853 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:15.334817 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-c11538-dockercfg-dpffn\"" Nov 03 21:30:15.389705 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:15.389662 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c90b432-31d5-49c0-8f4e-11397476f868-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-c11538-d9df47586-gj7tm\" (UID: \"5c90b432-31d5-49c0-8f4e-11397476f868\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c11538-d9df47586-gj7tm" Nov 03 21:30:15.389705 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:15.389708 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdfkf\" (UniqueName: \"kubernetes.io/projected/5c90b432-31d5-49c0-8f4e-11397476f868-kube-api-access-kdfkf\") pod \"payload-processing-e2e-shared-a-w0-c11538-d9df47586-gj7tm\" (UID: \"5c90b432-31d5-49c0-8f4e-11397476f868\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c11538-d9df47586-gj7tm" Nov 03 21:30:15.390487 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:15.390459 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c90b432-31d5-49c0-8f4e-11397476f868-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-c11538-d9df47586-gj7tm\" (UID: \"5c90b432-31d5-49c0-8f4e-11397476f868\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c11538-d9df47586-gj7tm" Nov 03 21:30:15.448440 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:15.448404 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdfkf\" (UniqueName: \"kubernetes.io/projected/5c90b432-31d5-49c0-8f4e-11397476f868-kube-api-access-kdfkf\") pod \"payload-processing-e2e-shared-a-w0-c11538-d9df47586-gj7tm\" (UID: \"5c90b432-31d5-49c0-8f4e-11397476f868\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c11538-d9df47586-gj7tm" Nov 03 21:30:15.594068 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:15.594034 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c11538-d9df47586-gj7tm" Nov 03 21:30:15.965035 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:15.965002 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-c11538-d9df47586-gj7tm"] Nov 03 21:30:15.966430 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:30:15.966395 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c90b432_31d5_49c0_8f4e_11397476f868.slice/crio-ae912b5668d7966c10084c7d211d3761b85fe8af3127e276ead5057c73282ce1 WatchSource:0}: Error finding container ae912b5668d7966c10084c7d211d3761b85fe8af3127e276ead5057c73282ce1: Status 404 returned error can't find the container with id ae912b5668d7966c10084c7d211d3761b85fe8af3127e276ead5057c73282ce1 Nov 03 21:30:15.999172 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:15.999135 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c11538-d9df47586-gj7tm" event={"ID":"5c90b432-31d5-49c0-8f4e-11397476f868","Type":"ContainerStarted","Data":"ae912b5668d7966c10084c7d211d3761b85fe8af3127e276ead5057c73282ce1"} Nov 03 21:30:17.014805 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:17.014759 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c11538-d9df47586-gj7tm" event={"ID":"5c90b432-31d5-49c0-8f4e-11397476f868","Type":"ContainerStarted","Data":"f703584d4f1b329553ae2e4e438c1ca884ed72e9c5e571f1c8e21c7414a50947"} Nov 03 21:30:17.015283 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:17.014984 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c11538-d9df47586-gj7tm" Nov 03 21:30:17.127211 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:17.127159 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c11538-d9df47586-gj7tm" podStartSLOduration=2.127144484 podStartE2EDuration="2.127144484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:17.125185769 +0000 UTC m=+2511.592964493" watchObservedRunningTime="2026-11-03 21:30:17.127144484 +0000 UTC m=+2511.594923242" Nov 03 21:30:23.535204 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:23.535166 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8"] Nov 03 21:30:23.539158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:23.539134 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8" Nov 03 21:30:23.589296 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:23.589264 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-c70fca-serving-cert\"" Nov 03 21:30:23.597094 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:23.597043 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8"] Nov 03 21:30:23.695618 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:23.695579 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef206c3a-762d-450d-9fb5-5a537392bffc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8\" (UID: \"ef206c3a-762d-450d-9fb5-5a537392bffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8" Nov 03 21:30:23.695792 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:23.695744 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef206c3a-762d-450d-9fb5-5a537392bffc-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8\" (UID: \"ef206c3a-762d-450d-9fb5-5a537392bffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8" Nov 03 21:30:23.695792 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:23.695783 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef206c3a-762d-450d-9fb5-5a537392bffc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8\" (UID: \"ef206c3a-762d-450d-9fb5-5a537392bffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8" Nov 03 21:30:23.695902 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:23.695883 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rr46\" (UniqueName: \"kubernetes.io/projected/ef206c3a-762d-450d-9fb5-5a537392bffc-kube-api-access-9rr46\") pod \"maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8\" (UID: \"ef206c3a-762d-450d-9fb5-5a537392bffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8" Nov 03 21:30:23.808210 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:23.806095 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef206c3a-762d-450d-9fb5-5a537392bffc-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8\" (UID: \"ef206c3a-762d-450d-9fb5-5a537392bffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8" Nov 03 21:30:23.808210 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:23.806165 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef206c3a-762d-450d-9fb5-5a537392bffc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8\" (UID: \"ef206c3a-762d-450d-9fb5-5a537392bffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8" Nov 03 21:30:23.808210 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:23.806244 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rr46\" (UniqueName: \"kubernetes.io/projected/ef206c3a-762d-450d-9fb5-5a537392bffc-kube-api-access-9rr46\") pod \"maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8\" (UID: \"ef206c3a-762d-450d-9fb5-5a537392bffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8" Nov 03 21:30:23.808210 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:23.806425 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef206c3a-762d-450d-9fb5-5a537392bffc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8\" (UID: \"ef206c3a-762d-450d-9fb5-5a537392bffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8" Nov 03 21:30:23.808210 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:23.808096 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef206c3a-762d-450d-9fb5-5a537392bffc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8\" (UID: \"ef206c3a-762d-450d-9fb5-5a537392bffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8" Nov 03 21:30:23.834116 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:23.831966 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef206c3a-762d-450d-9fb5-5a537392bffc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8\" (UID: \"ef206c3a-762d-450d-9fb5-5a537392bffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8" Nov 03 21:30:23.843478 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:23.843438 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef206c3a-762d-450d-9fb5-5a537392bffc-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8\" (UID: \"ef206c3a-762d-450d-9fb5-5a537392bffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8" Nov 03 21:30:23.866485 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:23.866443 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rr46\" (UniqueName: \"kubernetes.io/projected/ef206c3a-762d-450d-9fb5-5a537392bffc-kube-api-access-9rr46\") pod \"maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8\" (UID: \"ef206c3a-762d-450d-9fb5-5a537392bffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8" Nov 03 21:30:24.153703 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:24.153616 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8" Nov 03 21:30:24.531636 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:30:24.531595 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef206c3a_762d_450d_9fb5_5a537392bffc.slice/crio-e56a5aae820478918031e8d96349757cbbb0885717d0d498d39c67eb8be284ac WatchSource:0}: Error finding container e56a5aae820478918031e8d96349757cbbb0885717d0d498d39c67eb8be284ac: Status 404 returned error can't find the container with id e56a5aae820478918031e8d96349757cbbb0885717d0d498d39c67eb8be284ac Nov 03 21:30:24.535980 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:24.535481 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8"] Nov 03 21:30:25.067654 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:25.067610 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8" event={"ID":"ef206c3a-762d-450d-9fb5-5a537392bffc","Type":"ContainerStarted","Data":"e56a5aae820478918031e8d96349757cbbb0885717d0d498d39c67eb8be284ac"} Nov 03 21:30:26.107338 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:26.107294 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8" event={"ID":"ef206c3a-762d-450d-9fb5-5a537392bffc","Type":"ContainerStarted","Data":"ce4d1e37084c7a1448bda70c49ffa2b9c562346a3f5b69a0fe2298b4a51f7fa7"} Nov 03 21:30:26.108371 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:26.108350 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8" Nov 03 21:30:26.221575 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:26.221507 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8" podStartSLOduration=2.288493994 podStartE2EDuration="3.221492153s" totalImagesPullingTime="932.998159ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:26.220409806 +0000 UTC m=+2520.688188523" watchObservedRunningTime="2026-11-03 21:30:26.221492153 +0000 UTC m=+2520.689270875" Nov 03 21:30:26.847793 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:26.847757 5615 scope.go:122] "RemoveContainer" containerID="29e687e739ace9ba589b2e71fe10b2953eabb7f2c63056827d5357ef263c9cb9" Nov 03 21:30:28.047099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:28.046505 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c11538-d9df47586-gj7tm" Nov 03 21:30:33.159625 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:33.159588 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8" Nov 03 21:30:40.619709 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:30:40.619619 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:30:42.175247 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:42.175190 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h"] Nov 03 21:30:42.175838 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:42.175559 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h" podUID="9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a" containerName="maas-api" containerID="cri-o://a0121807123bb57235525e54d0458f11ec21e91c7a88ad75be84a887b929a5a6" gracePeriod=30 Nov 03 21:30:43.448147 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:43.448114 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h" Nov 03 21:30:43.521289 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:43.521209 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-kube-api-access-km9np\" (UniqueName: \"kubernetes.io/projected/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-kube-api-access-km9np\") pod \"9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a\" (UID: \"9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a\") " Nov 03 21:30:43.521441 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:43.521343 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-openshift-service-ca\") pod \"9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a\" (UID: \"9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a\") " Nov 03 21:30:43.521441 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:43.521363 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-maas-api-tls\") pod \"9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a\" (UID: \"9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a\") " Nov 03 21:30:43.521441 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:43.521383 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-metrics-certs\") pod \"9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a\" (UID: \"9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a\") " Nov 03 21:30:43.521753 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:43.521726 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-openshift-service-ca" pod "9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a" (UID: "9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:30:43.523507 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:43.523478 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-metrics-certs" pod "9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a" (UID: "9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:30:43.523606 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:43.523571 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-kube-api-access-km9np" pod "9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a" (UID: "9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a"). InnerVolumeSpecName "kube-api-access-km9np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:30:43.523646 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:43.523622 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-maas-api-tls" pod "9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a" (UID: "9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:30:43.622669 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:43.622630 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:30:43.622669 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:43.622664 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:30:43.622669 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:43.622674 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:30:43.622917 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:43.622684 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km9np\" (UniqueName: \"kubernetes.io/projected/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a-kube-api-access-km9np\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:30:44.282280 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:44.282241 5615 generic.go:356] "Generic (PLEG): container finished" podID="9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a" containerID="a0121807123bb57235525e54d0458f11ec21e91c7a88ad75be84a887b929a5a6" exitCode=0 Nov 03 21:30:44.282458 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:44.282430 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h" event={"ID":"9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a","Type":"ContainerDied","Data":"a0121807123bb57235525e54d0458f11ec21e91c7a88ad75be84a887b929a5a6"} Nov 03 21:30:44.282507 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:44.282480 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h" event={"ID":"9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a","Type":"ContainerDied","Data":"5693ca1728162b10f23e66aa103010093ade37f44516dbe1633b2993ef537528"} Nov 03 21:30:44.282547 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:44.282508 5615 scope.go:122] "RemoveContainer" containerID="a0121807123bb57235525e54d0458f11ec21e91c7a88ad75be84a887b929a5a6" Nov 03 21:30:44.282677 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:44.282662 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h" Nov 03 21:30:44.294161 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:44.294069 5615 scope.go:122] "RemoveContainer" containerID="a0121807123bb57235525e54d0458f11ec21e91c7a88ad75be84a887b929a5a6" Nov 03 21:30:44.294413 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:30:44.294386 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0121807123bb57235525e54d0458f11ec21e91c7a88ad75be84a887b929a5a6\": container with ID starting with a0121807123bb57235525e54d0458f11ec21e91c7a88ad75be84a887b929a5a6 not found: ID does not exist" containerID="a0121807123bb57235525e54d0458f11ec21e91c7a88ad75be84a887b929a5a6" Nov 03 21:30:44.294466 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:44.294425 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0121807123bb57235525e54d0458f11ec21e91c7a88ad75be84a887b929a5a6"} err="failed to get container status \"a0121807123bb57235525e54d0458f11ec21e91c7a88ad75be84a887b929a5a6\": rpc error: code = NotFound desc = could not find container \"a0121807123bb57235525e54d0458f11ec21e91c7a88ad75be84a887b929a5a6\": container with ID starting with a0121807123bb57235525e54d0458f11ec21e91c7a88ad75be84a887b929a5a6 not found: ID does not exist" Nov 03 21:30:44.440269 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:44.440232 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h"] Nov 03 21:30:44.487152 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:44.487113 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3d317d-8d988b5d4-55w6h"] Nov 03 21:30:45.652253 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:45.652218 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a" path="/var/lib/kubelet/pods/9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a/volumes" Nov 03 21:30:56.435957 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:56.435866 5615 generic.go:356] "Generic (PLEG): container finished" podID="782d9d6a-583d-4bdc-af1e-d87d81db3d4a" containerID="aa5d5e70a16b8e2a7c9bc5b90e58a014962e173d9931e48f17f47cd709d08ca2" exitCode=0 Nov 03 21:30:56.435957 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:56.435932 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" event={"ID":"782d9d6a-583d-4bdc-af1e-d87d81db3d4a","Type":"ContainerDied","Data":"aa5d5e70a16b8e2a7c9bc5b90e58a014962e173d9931e48f17f47cd709d08ca2"} Nov 03 21:30:56.436534 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:56.436405 5615 scope.go:122] "RemoveContainer" containerID="aa5d5e70a16b8e2a7c9bc5b90e58a014962e173d9931e48f17f47cd709d08ca2" Nov 03 21:30:56.437868 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:56.437842 5615 generic.go:356] "Generic (PLEG): container finished" podID="88203e05-6890-4c96-9d15-07c6685ad6d3" containerID="1b9eab921b1dfd61588c0517bc335ac9509661386f69c4ff6c9398c4f29cc7ca" exitCode=0 Nov 03 21:30:56.437960 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:56.437874 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" event={"ID":"88203e05-6890-4c96-9d15-07c6685ad6d3","Type":"ContainerDied","Data":"1b9eab921b1dfd61588c0517bc335ac9509661386f69c4ff6c9398c4f29cc7ca"} Nov 03 21:30:56.438254 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:56.438224 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:30:56.438309 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:56.438286 5615 scope.go:122] "RemoveContainer" containerID="1b9eab921b1dfd61588c0517bc335ac9509661386f69c4ff6c9398c4f29cc7ca" Nov 03 21:30:56.438345 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:56.438322 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:30:56.438390 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:56.438374 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:30:56.439947 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:56.439913 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:30:56.440048 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:56.439993 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:30:56.440048 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:56.440040 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:30:57.458261 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:57.458220 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" event={"ID":"782d9d6a-583d-4bdc-af1e-d87d81db3d4a","Type":"ContainerStarted","Data":"b34cd6395e5acac9322bb639c7107b94cc6ad056e87677c505e4ac5859799e93"} Nov 03 21:30:57.463359 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:57.463327 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" event={"ID":"88203e05-6890-4c96-9d15-07c6685ad6d3","Type":"ContainerStarted","Data":"6b8a9aa0533594d108308e2e5a31ffe7afad7e8c33468d2e1e996ff0bb634ac1"} Nov 03 21:30:57.464412 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:57.464385 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8"] Nov 03 21:30:57.465060 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:57.465038 5615 cpu_manager.go:414] "containerMap: removing container" podUID="9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a" containerName="maas-api" Nov 03 21:30:57.465183 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:57.465061 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a" containerName="maas-api" Nov 03 21:30:57.465268 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:57.465255 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b5d0fab-d0db-4bcd-94e6-8bfea3d2d14a" containerName="maas-api" Nov 03 21:30:57.468033 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:57.468017 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8" Nov 03 21:30:57.482282 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:57.482256 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8\" (UID: \"98dbffe1-fe8a-43fa-821a-cfe49074a6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8" Nov 03 21:30:57.482393 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:57.482290 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlbc7\" (UniqueName: \"kubernetes.io/projected/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-kube-api-access-hlbc7\") pod \"maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8\" (UID: \"98dbffe1-fe8a-43fa-821a-cfe49074a6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8" Nov 03 21:30:57.482393 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:57.482358 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8\" (UID: \"98dbffe1-fe8a-43fa-821a-cfe49074a6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8" Nov 03 21:30:57.482514 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:57.482493 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8\" (UID: \"98dbffe1-fe8a-43fa-821a-cfe49074a6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8" Nov 03 21:30:57.529199 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:57.529159 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-c11538-serving-cert\"" Nov 03 21:30:57.549139 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:57.548244 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8"] Nov 03 21:30:57.583559 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:57.583528 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8\" (UID: \"98dbffe1-fe8a-43fa-821a-cfe49074a6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8" Nov 03 21:30:57.583769 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:57.583682 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlbc7\" (UniqueName: \"kubernetes.io/projected/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-kube-api-access-hlbc7\") pod \"maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8\" (UID: \"98dbffe1-fe8a-43fa-821a-cfe49074a6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8" Nov 03 21:30:57.583769 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:30:57.583698 5615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-serving-cert: secret "maas-api-e2e-shared-a-w0-c11538-serving-cert" not found Nov 03 21:30:57.583902 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:30:57.583778 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-maas-api-tls podName:98dbffe1-fe8a-43fa-821a-cfe49074a6a9 nodeName:}" failed. No retries permitted until 2026-11-03 21:30:58.083754584 +0000 UTC m=+2552.551533287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-maas-api-tls") pod "maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8" (UID: "98dbffe1-fe8a-43fa-821a-cfe49074a6a9") : secret "maas-api-e2e-shared-a-w0-c11538-serving-cert" not found Nov 03 21:30:57.583902 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:57.583803 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8\" (UID: \"98dbffe1-fe8a-43fa-821a-cfe49074a6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8" Nov 03 21:30:57.584018 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:57.583931 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8\" (UID: \"98dbffe1-fe8a-43fa-821a-cfe49074a6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8" Nov 03 21:30:57.584893 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:57.584870 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8\" (UID: \"98dbffe1-fe8a-43fa-821a-cfe49074a6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8" Nov 03 21:30:57.589108 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:57.587888 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8\" (UID: \"98dbffe1-fe8a-43fa-821a-cfe49074a6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8" Nov 03 21:30:57.654184 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:57.653933 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlbc7\" (UniqueName: \"kubernetes.io/projected/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-kube-api-access-hlbc7\") pod \"maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8\" (UID: \"98dbffe1-fe8a-43fa-821a-cfe49074a6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8" Nov 03 21:30:57.657172 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:57.656110 5615 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 21:30:57.657172 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:57.656169 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" podUID="88203e05-6890-4c96-9d15-07c6685ad6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 21:30:57.665594 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:57.665515 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:30:58.095037 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:58.094999 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8\" (UID: \"98dbffe1-fe8a-43fa-821a-cfe49074a6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8" Nov 03 21:30:58.135411 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:58.135378 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8\" (UID: \"98dbffe1-fe8a-43fa-821a-cfe49074a6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8" Nov 03 21:30:58.232099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:58.231471 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:30:58.232099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:58.231603 5615 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 21:30:58.232099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:58.231649 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" podUID="782d9d6a-583d-4bdc-af1e-d87d81db3d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 21:30:58.386370 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:58.386280 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8" Nov 03 21:30:58.651303 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:58.650729 5615 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 21:30:58.651303 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:58.650803 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" podUID="88203e05-6890-4c96-9d15-07c6685ad6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 21:30:58.849183 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:30:58.848838 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98dbffe1_fe8a_43fa_821a_cfe49074a6a9.slice/crio-93bfd54cb7183d9b957570e35b1a8fff28131d80e643cf3f1c699de130affffe WatchSource:0}: Error finding container 93bfd54cb7183d9b957570e35b1a8fff28131d80e643cf3f1c699de130affffe: Status 404 returned error can't find the container with id 93bfd54cb7183d9b957570e35b1a8fff28131d80e643cf3f1c699de130affffe Nov 03 21:30:58.851222 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:58.850300 5615 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:30:58.852601 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:58.852425 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8"] Nov 03 21:30:59.229099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:59.228788 5615 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 21:30:59.229099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:59.228851 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" podUID="782d9d6a-583d-4bdc-af1e-d87d81db3d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 21:30:59.481195 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:59.481109 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8" event={"ID":"98dbffe1-fe8a-43fa-821a-cfe49074a6a9","Type":"ContainerStarted","Data":"93bfd54cb7183d9b957570e35b1a8fff28131d80e643cf3f1c699de130affffe"} Nov 03 21:30:59.545812 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:59.545732 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:30:59.627548 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:59.627507 5615 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 21:30:59.627827 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:30:59.627799 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" podUID="88203e05-6890-4c96-9d15-07c6685ad6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 21:31:00.249100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:00.248576 5615 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 21:31:00.249100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:00.248643 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" podUID="782d9d6a-583d-4bdc-af1e-d87d81db3d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 21:31:00.560159 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:00.559533 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8" event={"ID":"98dbffe1-fe8a-43fa-821a-cfe49074a6a9","Type":"ContainerStarted","Data":"b3b750da0a6dd7410cf5146d9ceb9b4c4ca2b9b3b352cb94db78e728cc203930"} Nov 03 21:31:00.562104 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:00.561070 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8" Nov 03 21:31:00.626553 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:00.626511 5615 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 21:31:00.626866 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:00.626830 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" podUID="88203e05-6890-4c96-9d15-07c6685ad6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 21:31:00.684746 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:00.684682 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8" podStartSLOduration=2.841222244 podStartE2EDuration="3.684667038s" totalImagesPullingTime="843.444794ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:31:00.683667565 +0000 UTC m=+2555.151446288" watchObservedRunningTime="2026-11-03 21:31:00.684667038 +0000 UTC m=+2555.152445761" Nov 03 21:31:01.246100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:01.245557 5615 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 21:31:01.246100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:01.245646 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" podUID="782d9d6a-583d-4bdc-af1e-d87d81db3d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 21:31:01.646721 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:01.646627 5615 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 21:31:01.647335 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:01.647301 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" podUID="88203e05-6890-4c96-9d15-07c6685ad6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 21:31:01.733414 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:01.733367 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr"] Nov 03 21:31:01.733909 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:01.733871 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr" podUID="8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a" containerName="maas-api" containerID="cri-o://87f4374e9ebbb843edcca4180cea9aa7c60f3ba49baebc28dddf31f5c6f1d494" gracePeriod=30 Nov 03 21:31:02.270101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:02.246391 5615 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 21:31:02.270101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:02.246452 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" podUID="782d9d6a-583d-4bdc-af1e-d87d81db3d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 21:31:02.390903 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:02.390874 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr" Nov 03 21:31:02.455756 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:02.455702 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-maas-api-tls\") pod \"8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a\" (UID: \"8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a\") " Nov 03 21:31:02.455958 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:02.455867 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-openshift-service-ca\") pod \"8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a\" (UID: \"8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a\") " Nov 03 21:31:02.456640 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:02.456600 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-openshift-service-ca" pod "8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a" (UID: "8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:31:02.458614 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:02.458544 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-maas-api-tls" pod "8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a" (UID: "8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:31:02.557515 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:02.557478 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-metrics-certs\") pod \"8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a\" (UID: \"8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a\") " Nov 03 21:31:02.557747 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:02.557732 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-kube-api-access-ftnqz\" (UniqueName: \"kubernetes.io/projected/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-kube-api-access-ftnqz\") pod \"8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a\" (UID: \"8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a\") " Nov 03 21:31:02.558359 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:02.558339 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:31:02.558461 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:02.558451 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:31:02.559940 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:02.559913 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-metrics-certs" pod "8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a" (UID: "8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:31:02.560438 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:02.560418 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-kube-api-access-ftnqz" pod "8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a" (UID: "8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a"). InnerVolumeSpecName "kube-api-access-ftnqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:31:02.575233 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:02.575205 5615 generic.go:356] "Generic (PLEG): container finished" podID="8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a" containerID="87f4374e9ebbb843edcca4180cea9aa7c60f3ba49baebc28dddf31f5c6f1d494" exitCode=0 Nov 03 21:31:02.576182 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:02.576148 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr" Nov 03 21:31:02.576389 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:02.576213 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr" event={"ID":"8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a","Type":"ContainerDied","Data":"87f4374e9ebbb843edcca4180cea9aa7c60f3ba49baebc28dddf31f5c6f1d494"} Nov 03 21:31:02.576511 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:02.576496 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr" event={"ID":"8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a","Type":"ContainerDied","Data":"c903c341a4947063c69db283f3b8fd7770b1fdb945ce73cc4c7295ccf1ed4a79"} Nov 03 21:31:02.576614 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:02.576565 5615 scope.go:122] "RemoveContainer" containerID="87f4374e9ebbb843edcca4180cea9aa7c60f3ba49baebc28dddf31f5c6f1d494" Nov 03 21:31:02.587153 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:02.587134 5615 scope.go:122] "RemoveContainer" containerID="87f4374e9ebbb843edcca4180cea9aa7c60f3ba49baebc28dddf31f5c6f1d494" Nov 03 21:31:02.587537 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:31:02.587512 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87f4374e9ebbb843edcca4180cea9aa7c60f3ba49baebc28dddf31f5c6f1d494\": container with ID starting with 87f4374e9ebbb843edcca4180cea9aa7c60f3ba49baebc28dddf31f5c6f1d494 not found: ID does not exist" containerID="87f4374e9ebbb843edcca4180cea9aa7c60f3ba49baebc28dddf31f5c6f1d494" Nov 03 21:31:02.587607 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:02.587543 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87f4374e9ebbb843edcca4180cea9aa7c60f3ba49baebc28dddf31f5c6f1d494"} err="failed to get container status \"87f4374e9ebbb843edcca4180cea9aa7c60f3ba49baebc28dddf31f5c6f1d494\": rpc error: code = NotFound desc = could not find container \"87f4374e9ebbb843edcca4180cea9aa7c60f3ba49baebc28dddf31f5c6f1d494\": container with ID starting with 87f4374e9ebbb843edcca4180cea9aa7c60f3ba49baebc28dddf31f5c6f1d494 not found: ID does not exist" Nov 03 21:31:02.642221 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:02.641152 5615 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 21:31:02.642221 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:02.641214 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" podUID="88203e05-6890-4c96-9d15-07c6685ad6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 21:31:02.660709 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:02.660676 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:31:02.661164 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:02.660715 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftnqz\" (UniqueName: \"kubernetes.io/projected/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a-kube-api-access-ftnqz\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:31:02.778760 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:02.775281 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr"] Nov 03 21:31:02.786488 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:02.785128 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fe2df9-5fbff6ff65-7hrkr"] Nov 03 21:31:03.233494 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:03.233457 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:31:03.233946 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:03.233924 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:31:03.234827 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:03.234810 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 21:31:03.329384 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:03.329346 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b"] Nov 03 21:31:03.330157 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:03.330135 5615 cpu_manager.go:414] "containerMap: removing container" podUID="8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a" containerName="maas-api" Nov 03 21:31:03.330157 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:03.330155 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a" containerName="maas-api" Nov 03 21:31:03.330320 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:03.330279 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a" containerName="maas-api" Nov 03 21:31:03.333609 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:03.333580 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b" Nov 03 21:31:03.374387 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:03.374352 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b\" (UID: \"190b83fb-12d3-4fa6-97f8-bfdb76990fe8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b" Nov 03 21:31:03.374387 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:03.374392 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq7bw\" (UniqueName: \"kubernetes.io/projected/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-kube-api-access-bq7bw\") pod \"maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b\" (UID: \"190b83fb-12d3-4fa6-97f8-bfdb76990fe8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b" Nov 03 21:31:03.374607 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:03.374416 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-metrics-certs\") pod \"maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b\" (UID: \"190b83fb-12d3-4fa6-97f8-bfdb76990fe8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b" Nov 03 21:31:03.374607 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:03.374534 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b\" (UID: \"190b83fb-12d3-4fa6-97f8-bfdb76990fe8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b" Nov 03 21:31:03.384158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:03.384131 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-cc4de5-serving-cert\"" Nov 03 21:31:03.389807 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:03.389779 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b"] Nov 03 21:31:03.475971 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:03.475932 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b\" (UID: \"190b83fb-12d3-4fa6-97f8-bfdb76990fe8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b" Nov 03 21:31:03.476194 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:03.475986 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq7bw\" (UniqueName: \"kubernetes.io/projected/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-kube-api-access-bq7bw\") pod \"maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b\" (UID: \"190b83fb-12d3-4fa6-97f8-bfdb76990fe8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b" Nov 03 21:31:03.476194 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:03.476024 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-metrics-certs\") pod \"maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b\" (UID: \"190b83fb-12d3-4fa6-97f8-bfdb76990fe8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b" Nov 03 21:31:03.476194 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:03.476087 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b\" (UID: \"190b83fb-12d3-4fa6-97f8-bfdb76990fe8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b" Nov 03 21:31:03.476194 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:31:03.476142 5615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-serving-cert: secret "maas-api-e2e-praxis-b-cc4de5-serving-cert" not found Nov 03 21:31:03.476417 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:31:03.476216 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-maas-api-tls podName:190b83fb-12d3-4fa6-97f8-bfdb76990fe8 nodeName:}" failed. No retries permitted until 2026-11-03 21:31:03.976194557 +0000 UTC m=+2558.443973275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-maas-api-tls") pod "maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b" (UID: "190b83fb-12d3-4fa6-97f8-bfdb76990fe8") : secret "maas-api-e2e-praxis-b-cc4de5-serving-cert" not found Nov 03 21:31:03.476883 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:03.476857 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b\" (UID: \"190b83fb-12d3-4fa6-97f8-bfdb76990fe8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b" Nov 03 21:31:03.479227 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:03.479193 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-metrics-certs\") pod \"maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b\" (UID: \"190b83fb-12d3-4fa6-97f8-bfdb76990fe8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b" Nov 03 21:31:03.535097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:03.534998 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq7bw\" (UniqueName: \"kubernetes.io/projected/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-kube-api-access-bq7bw\") pod \"maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b\" (UID: \"190b83fb-12d3-4fa6-97f8-bfdb76990fe8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b" Nov 03 21:31:03.630663 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:03.630621 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:31:03.631790 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:03.631765 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 21:31:03.739686 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:03.738502 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a" path="/var/lib/kubelet/pods/8cfa7b2e-e2b8-435e-94dc-37fb2fdd469a/volumes" Nov 03 21:31:04.001097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:04.000425 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b\" (UID: \"190b83fb-12d3-4fa6-97f8-bfdb76990fe8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b" Nov 03 21:31:04.110433 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:04.110394 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b\" (UID: \"190b83fb-12d3-4fa6-97f8-bfdb76990fe8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b" Nov 03 21:31:04.246965 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:04.246922 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b" Nov 03 21:31:04.621059 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:04.621026 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b"] Nov 03 21:31:04.622399 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:31:04.622372 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod190b83fb_12d3_4fa6_97f8_bfdb76990fe8.slice/crio-f8e79f853d68463951600f76e79334610aa5993ae8e3cefc37037d309fd9b9c5 WatchSource:0}: Error finding container f8e79f853d68463951600f76e79334610aa5993ae8e3cefc37037d309fd9b9c5: Status 404 returned error can't find the container with id f8e79f853d68463951600f76e79334610aa5993ae8e3cefc37037d309fd9b9c5 Nov 03 21:31:05.599146 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:05.599066 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b" event={"ID":"190b83fb-12d3-4fa6-97f8-bfdb76990fe8","Type":"ContainerStarted","Data":"f2535f4e07e36c4fb454d4f52a46ebb59285c1b5db912d6e0e20cf686fc1e5f2"} Nov 03 21:31:05.599591 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:05.599163 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b" Nov 03 21:31:05.599591 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:05.599216 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b" event={"ID":"190b83fb-12d3-4fa6-97f8-bfdb76990fe8","Type":"ContainerStarted","Data":"f8e79f853d68463951600f76e79334610aa5993ae8e3cefc37037d309fd9b9c5"} Nov 03 21:31:05.751927 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:05.751863 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b" podStartSLOduration=1.8525162910000001 podStartE2EDuration="2.751848229s" totalImagesPullingTime="899.331938ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:31:05.749333394 +0000 UTC m=+2560.217112116" watchObservedRunningTime="2026-11-03 21:31:05.751848229 +0000 UTC m=+2560.219626987" Nov 03 21:31:07.582973 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:07.582943 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8" Nov 03 21:31:10.620240 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:31:10.620199 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:31:11.612495 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:11.612460 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b" Nov 03 21:31:14.550785 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:14.550743 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw"] Nov 03 21:31:14.551303 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:14.550977 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" podUID="69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87" containerName="istio-proxy" containerID="cri-o://e4096b6622a61dff89ca87353aef901d1b3f5de38edb3e69bd2eb56fab72a87b" gracePeriod=30 Nov 03 21:31:19.745601 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.745556 5615 generic.go:356] "Generic (PLEG): container finished" podID="69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87" containerID="e4096b6622a61dff89ca87353aef901d1b3f5de38edb3e69bd2eb56fab72a87b" exitCode=0 Nov 03 21:31:19.746059 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.745779 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" event={"ID":"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87","Type":"ContainerDied","Data":"e4096b6622a61dff89ca87353aef901d1b3f5de38edb3e69bd2eb56fab72a87b"} Nov 03 21:31:19.746059 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.745881 5615 scope.go:122] "RemoveContainer" containerID="37c59b60e2c16777df037693df6ddad73ffb754cd04ed153ec359cd37fcf4ee0" Nov 03 21:31:19.832956 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.832883 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:31:19.868013 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.867979 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-kube-api-access-k4jl7\" (UniqueName: \"kubernetes.io/projected/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-kube-api-access-k4jl7\") pod \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " Nov 03 21:31:19.868210 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.868019 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-workload-certs\") pod \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " Nov 03 21:31:19.868210 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.868062 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istio-envoy\") pod \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " Nov 03 21:31:19.868210 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.868129 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istio-data\") pod \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " Nov 03 21:31:19.868771 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.868253 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istio-podinfo\") pod \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " Nov 03 21:31:19.868771 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.868284 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-workload-socket\") pod \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " Nov 03 21:31:19.868771 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.868361 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-credential-socket\") pod \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " Nov 03 21:31:19.868771 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.868391 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-workload-certs" pod "69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87" (UID: "69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:31:19.868771 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.868414 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istio-token\" (UniqueName: \"kubernetes.io/projected/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istio-token\") pod \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " Nov 03 21:31:19.868771 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.868433 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istio-data" pod "69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87" (UID: "69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:31:19.868771 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.868550 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istiod-ca-cert\") pod \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\" (UID: \"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87\") " Nov 03 21:31:19.868771 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.868568 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-workload-socket" pod "69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87" (UID: "69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:31:19.868771 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.868639 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-credential-socket" pod "69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87" (UID: "69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:31:19.869282 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.868976 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istiod-ca-cert" pod "69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87" (UID: "69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:31:19.869282 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.869113 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:31:19.869282 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.869130 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:31:19.869282 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.869140 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:31:19.869282 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.869150 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:31:19.869282 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.869159 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:31:19.870661 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.870633 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istio-podinfo" pod "69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87" (UID: "69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:31:19.870763 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.870735 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istio-token" pod "69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87" (UID: "69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:31:19.870810 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.870779 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-kube-api-access-k4jl7" pod "69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87" (UID: "69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87"). InnerVolumeSpecName "kube-api-access-k4jl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:31:19.870847 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.870834 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istio-envoy" pod "69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87" (UID: "69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:31:19.970038 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.969995 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:31:19.970038 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.970036 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:31:19.970301 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.970053 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:31:19.970301 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:19.970067 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4jl7\" (UniqueName: \"kubernetes.io/projected/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87-kube-api-access-k4jl7\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:31:20.753424 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:20.753391 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" Nov 03 21:31:20.753863 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:20.753430 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw" event={"ID":"69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87","Type":"ContainerDied","Data":"d0ba26a9c2dd6ab9d0558c8c602822a7cf69de133ee82fdb01b0735793322783"} Nov 03 21:31:20.753863 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:20.753492 5615 scope.go:122] "RemoveContainer" containerID="e4096b6622a61dff89ca87353aef901d1b3f5de38edb3e69bd2eb56fab72a87b" Nov 03 21:31:20.958101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:20.958048 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw"] Nov 03 21:31:21.005769 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:21.005680 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-fe2df9-openshift-default-bc77ff4d9-tccrw"] Nov 03 21:31:21.652433 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:21.652396 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87" path="/var/lib/kubelet/pods/69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87/volumes" Nov 03 21:31:38.849936 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:38.849896 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg"] Nov 03 21:31:38.851234 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:38.851208 5615 cpu_manager.go:414] "containerMap: removing container" podUID="69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87" containerName="istio-proxy" Nov 03 21:31:38.851234 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:38.851233 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87" containerName="istio-proxy" Nov 03 21:31:38.851441 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:38.851301 5615 cpu_manager.go:414] "containerMap: removing container" podUID="69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87" containerName="istio-proxy" Nov 03 21:31:38.851441 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:38.851311 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87" containerName="istio-proxy" Nov 03 21:31:38.851550 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:38.851473 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87" containerName="istio-proxy" Nov 03 21:31:38.889202 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:38.889166 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:38.941718 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:38.940862 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-a6219b-openshift-default-dockercfg-7mw42\"" Nov 03 21:31:39.018926 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.018823 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg"] Nov 03 21:31:39.070098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.054990 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89lmc\" (UniqueName: \"kubernetes.io/projected/0fd70b58-6f75-415c-b26e-303faf172e6c-kube-api-access-89lmc\") pod \"e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:39.070098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.055116 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-workload-socket\") pod \"e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:39.070098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.055149 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-workload-certs\") pod \"e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:39.070098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.055199 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-credential-socket\") pod \"e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:39.070098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.055292 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fd70b58-6f75-415c-b26e-303faf172e6c-istio-token\") pod \"e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:39.070098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.055359 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-istio-data\") pod \"e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:39.070098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.055426 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fd70b58-6f75-415c-b26e-303faf172e6c-istio-podinfo\") pod \"e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:39.070098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.055473 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fd70b58-6f75-415c-b26e-303faf172e6c-istiod-ca-cert\") pod \"e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:39.070098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.055513 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-istio-envoy\") pod \"e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:39.157296 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.157185 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89lmc\" (UniqueName: \"kubernetes.io/projected/0fd70b58-6f75-415c-b26e-303faf172e6c-kube-api-access-89lmc\") pod \"e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:39.157484 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.157295 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-workload-socket\") pod \"e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:39.157484 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.157331 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-workload-certs\") pod \"e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:39.157484 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.157378 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-credential-socket\") pod \"e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:39.157484 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.157472 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fd70b58-6f75-415c-b26e-303faf172e6c-istio-token\") pod \"e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:39.157767 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.157542 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-istio-data\") pod \"e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:39.157767 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.157619 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fd70b58-6f75-415c-b26e-303faf172e6c-istio-podinfo\") pod \"e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:39.157767 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.157670 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fd70b58-6f75-415c-b26e-303faf172e6c-istiod-ca-cert\") pod \"e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:39.157767 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.157712 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-istio-envoy\") pod \"e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:39.158004 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.157965 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-workload-certs\") pod \"e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:39.158302 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.158279 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-workload-socket\") pod \"e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:39.158444 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.158423 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-istio-data\") pod \"e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:39.159459 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.159086 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-credential-socket\") pod \"e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:39.159459 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.159418 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fd70b58-6f75-415c-b26e-303faf172e6c-istiod-ca-cert\") pod \"e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:39.160963 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.160934 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-istio-envoy\") pod \"e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:39.161145 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.161126 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fd70b58-6f75-415c-b26e-303faf172e6c-istio-podinfo\") pod \"e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:39.216318 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.216284 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fd70b58-6f75-415c-b26e-303faf172e6c-istio-token\") pod \"e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:39.216540 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.216515 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89lmc\" (UniqueName: \"kubernetes.io/projected/0fd70b58-6f75-415c-b26e-303faf172e6c-kube-api-access-89lmc\") pod \"e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:39.325150 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.323252 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:39.841472 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.837192 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg"] Nov 03 21:31:39.884505 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.884460 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:31:39.884937 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.884550 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:31:39.884937 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.884587 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:31:39.950300 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.950258 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" event={"ID":"0fd70b58-6f75-415c-b26e-303faf172e6c","Type":"ContainerStarted","Data":"745517029e148e61125c37d9b12a11aca05882669e018998498a48414bfc5a4b"} Nov 03 21:31:39.989315 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.989256 5615 patch_prober.go:29] interesting pod/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:31:39.989500 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:39.989342 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" podUID="618668eb-db73-4af8-967f-f73331c493f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:31:40.620146 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:31:40.620035 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:31:40.958054 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:40.958013 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" event={"ID":"0fd70b58-6f75-415c-b26e-303faf172e6c","Type":"ContainerStarted","Data":"8b98f3786960a3457eddfefdc91186507ff2528407e8354152811066dd3f544e"} Nov 03 21:31:40.959864 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:40.959841 5615 generic.go:356] "Generic (PLEG): container finished" podID="77818da7-0c48-434f-b2cb-68d47b3c2a33" containerID="be7edeb286d1dda785653b4dcfada1800ead4f1fc7961349c0f7246bd05b8f58" exitCode=0 Nov 03 21:31:40.959956 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:40.959875 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" event={"ID":"77818da7-0c48-434f-b2cb-68d47b3c2a33","Type":"ContainerDied","Data":"be7edeb286d1dda785653b4dcfada1800ead4f1fc7961349c0f7246bd05b8f58"} Nov 03 21:31:40.960268 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:40.960251 5615 scope.go:122] "RemoveContainer" containerID="be7edeb286d1dda785653b4dcfada1800ead4f1fc7961349c0f7246bd05b8f58" Nov 03 21:31:40.961805 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:40.961768 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:31:40.961909 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:40.961846 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:31:40.961909 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:40.961901 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:31:41.162894 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:41.162832 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" podStartSLOduration=3.162811353 podStartE2EDuration="3.162811353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:31:41.125524815 +0000 UTC m=+2595.593303537" watchObservedRunningTime="2026-11-03 21:31:41.162811353 +0000 UTC m=+2595.630590077" Nov 03 21:31:41.325708 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:41.325604 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:41.331923 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:41.331892 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:41.982527 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:41.982405 5615 generic.go:356] "Generic (PLEG): container finished" podID="618668eb-db73-4af8-967f-f73331c493f2" containerID="d18982066814a79e6da0455fccfcf4a09093d54b440c74083b43531b6929c5ed" exitCode=0 Nov 03 21:31:41.982978 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:41.982492 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" event={"ID":"618668eb-db73-4af8-967f-f73331c493f2","Type":"ContainerDied","Data":"d18982066814a79e6da0455fccfcf4a09093d54b440c74083b43531b6929c5ed"} Nov 03 21:31:41.983205 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:41.983187 5615 scope.go:122] "RemoveContainer" containerID="d18982066814a79e6da0455fccfcf4a09093d54b440c74083b43531b6929c5ed" Nov 03 21:31:41.984780 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:41.984751 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" event={"ID":"77818da7-0c48-434f-b2cb-68d47b3c2a33","Type":"ContainerStarted","Data":"f1f32e6c781ccbef7c25972505e586d06624ce0830692fff71321ec8006e0d00"} Nov 03 21:31:41.984864 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:41.984819 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:41.985992 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:41.985971 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:31:41.986158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:41.986136 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:31:41.986208 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:41.986197 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:31:41.986247 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:41.986229 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:31:42.058939 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:42.058901 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:31:42.059160 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:42.059134 5615 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 21:31:42.059229 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:42.059195 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" podUID="77818da7-0c48-434f-b2cb-68d47b3c2a33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 21:31:42.993237 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:42.993207 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" event={"ID":"618668eb-db73-4af8-967f-f73331c493f2","Type":"ContainerStarted","Data":"91b76ea1d242632b4fd940d3919d82fe541c0ce0074274bfc1905210e3c59e65"} Nov 03 21:31:43.058965 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:43.058933 5615 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 21:31:43.059179 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:43.058993 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" podUID="77818da7-0c48-434f-b2cb-68d47b3c2a33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 21:31:43.585825 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:43.585781 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:31:43.586060 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:43.586021 5615 patch_prober.go:29] interesting pod/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 03 21:31:43.586172 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:43.586099 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" podUID="618668eb-db73-4af8-967f-f73331c493f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 03 21:31:44.059876 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:44.059838 5615 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 21:31:44.060315 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:44.059913 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" podUID="77818da7-0c48-434f-b2cb-68d47b3c2a33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 21:31:44.586899 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:44.586859 5615 patch_prober.go:29] interesting pod/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 03 21:31:44.587158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:44.586910 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" podUID="618668eb-db73-4af8-967f-f73331c493f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 03 21:31:45.060112 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:45.059667 5615 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 21:31:45.060112 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:45.059728 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" podUID="77818da7-0c48-434f-b2cb-68d47b3c2a33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 21:31:45.601103 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:45.598962 5615 patch_prober.go:29] interesting pod/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 03 21:31:45.601103 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:45.599013 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" podUID="618668eb-db73-4af8-967f-f73331c493f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 03 21:31:45.860294 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:45.860196 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp"] Nov 03 21:31:45.861747 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:45.861716 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="69a9e3ce-28e8-4a93-a6aa-ad2073cb4b87" containerName="istio-proxy" Nov 03 21:31:45.865481 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:45.865457 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:45.941101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:45.940222 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-4d1799-openshift-default-dockercfg-ztntf\"" Nov 03 21:31:45.994818 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:45.994784 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp"] Nov 03 21:31:46.194560 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.194463 5615 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 21:31:46.195237 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.195133 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" podUID="77818da7-0c48-434f-b2cb-68d47b3c2a33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 21:31:46.276118 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.239052 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-workload-certs\") pod \"e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:46.276118 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.239341 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istio-envoy\") pod \"e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:46.276118 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.239395 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2d5z\" (UniqueName: \"kubernetes.io/projected/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-kube-api-access-b2d5z\") pod \"e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:46.276118 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.239593 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istio-token\") pod \"e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:46.276118 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.239664 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istio-podinfo\") pod \"e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:46.276118 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.239701 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-workload-socket\") pod \"e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:46.276118 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.239896 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istiod-ca-cert\") pod \"e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:46.276118 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.240033 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istio-data\") pod \"e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:46.276118 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.240227 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-credential-socket\") pod \"e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:46.370200 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.370151 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istiod-ca-cert\") pod \"e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:46.372688 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.372631 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istio-data\") pod \"e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:46.374325 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.374299 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-credential-socket\") pod \"e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:46.374559 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.374541 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-workload-certs\") pod \"e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:46.374732 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.374717 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istio-envoy\") pod \"e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:46.374859 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.374843 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2d5z\" (UniqueName: \"kubernetes.io/projected/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-kube-api-access-b2d5z\") pod \"e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:46.375008 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.374993 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istio-token\") pod \"e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:46.375205 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.375184 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istio-podinfo\") pod \"e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:46.375425 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.375401 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-workload-socket\") pod \"e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:46.376029 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.376006 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-workload-socket\") pod \"e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:46.376248 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.372384 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istiod-ca-cert\") pod \"e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:46.377015 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.376640 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istio-data\") pod \"e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:46.377507 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.377473 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-credential-socket\") pod \"e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:46.377925 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.377902 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-workload-certs\") pod \"e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:46.396866 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.396785 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istio-envoy\") pod \"e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:46.478152 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.475793 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istio-podinfo\") pod \"e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:46.485733 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.485691 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2d5z\" (UniqueName: \"kubernetes.io/projected/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-kube-api-access-b2d5z\") pod \"e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:46.486620 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.486596 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istio-token\") pod \"e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:46.586636 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.586591 5615 patch_prober.go:29] interesting pod/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 03 21:31:46.586848 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.586664 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" podUID="618668eb-db73-4af8-967f-f73331c493f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 03 21:31:46.786410 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:46.786347 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:47.059991 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:47.059478 5615 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 21:31:47.059991 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:47.059544 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" podUID="77818da7-0c48-434f-b2cb-68d47b3c2a33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 21:31:47.585535 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:47.585506 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp"] Nov 03 21:31:47.586736 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:47.585881 5615 patch_prober.go:29] interesting pod/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 03 21:31:47.586736 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:47.585956 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" podUID="618668eb-db73-4af8-967f-f73331c493f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 03 21:31:47.586847 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:31:47.586786 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2e19f3_34a2_4ab2_acdd_5da3cb8b1829.slice/crio-720aefc4a14536b3fd9caf952ee42eee41cf31fc51b161073fac367843cd7c51 WatchSource:0}: Error finding container 720aefc4a14536b3fd9caf952ee42eee41cf31fc51b161073fac367843cd7c51: Status 404 returned error can't find the container with id 720aefc4a14536b3fd9caf952ee42eee41cf31fc51b161073fac367843cd7c51 Nov 03 21:31:47.593960 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:47.593919 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:31:47.594090 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:47.594004 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:31:47.594090 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:47.594049 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:31:48.064649 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:48.064619 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:31:48.065019 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:48.064994 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:31:48.066116 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:48.066092 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 21:31:48.385841 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:48.385750 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" event={"ID":"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829","Type":"ContainerStarted","Data":"7e3e4956bfdeeb65e4b0e46439f7a10fde5a99e478ff9f1e4216ff398956183e"} Nov 03 21:31:48.385841 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:48.385814 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" event={"ID":"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829","Type":"ContainerStarted","Data":"720aefc4a14536b3fd9caf952ee42eee41cf31fc51b161073fac367843cd7c51"} Nov 03 21:31:48.586452 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:48.586415 5615 patch_prober.go:29] interesting pod/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 03 21:31:48.586852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:48.586468 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" podUID="618668eb-db73-4af8-967f-f73331c493f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 03 21:31:48.787800 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:48.787756 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:48.793791 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:48.793762 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:48.911507 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:48.911455 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" podStartSLOduration=3.911437701 podStartE2EDuration="3.911437701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:31:48.515383977 +0000 UTC m=+2602.983162702" watchObservedRunningTime="2026-11-03 21:31:48.911437701 +0000 UTC m=+2603.379216424" Nov 03 21:31:49.393139 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:49.393098 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:49.394000 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:49.393980 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:31:49.591184 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:49.591153 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:31:49.591670 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:49.591604 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:31:49.592461 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:31:49.592442 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:32:10.620508 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:32:10.620420 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:32:12.974679 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:12.974639 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a41883-5d94544d6b-snqv5"] Nov 03 21:32:13.096925 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:13.096868 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a41883-5cbdf8bb47-m697s"] Nov 03 21:32:17.975373 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:17.975321 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-a41883-5d94544d6b-snqv5" podUID="9df4016a-4b05-4252-8246-98e8c1e6cce4" containerName="payload-processing" containerID="cri-o://3235f5d0965066e4eb508533d78d7ede9e6ab23e47cca352fe9fce885c746b64" gracePeriod=25 Nov 03 21:32:18.098480 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.098429 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a41883-5cbdf8bb47-m697s" podUID="db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5" containerName="payload-pre-processing" containerID="cri-o://4e4b3851b4a49f6b1d4f73845fa8eb366d86229ab0f14296b4f732b68bf783b4" gracePeriod=25 Nov 03 21:32:18.252122 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.252088 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a41883-5d94544d6b-snqv5" Nov 03 21:32:18.338224 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.338190 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df4016a-4b05-4252-8246-98e8c1e6cce4-kube-api-access-sn8tv\" (UniqueName: \"kubernetes.io/projected/9df4016a-4b05-4252-8246-98e8c1e6cce4-kube-api-access-sn8tv\") pod \"9df4016a-4b05-4252-8246-98e8c1e6cce4\" (UID: \"9df4016a-4b05-4252-8246-98e8c1e6cce4\") " Nov 03 21:32:18.338404 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.338357 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9df4016a-4b05-4252-8246-98e8c1e6cce4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9df4016a-4b05-4252-8246-98e8c1e6cce4-plugins-config-volume\") pod \"9df4016a-4b05-4252-8246-98e8c1e6cce4\" (UID: \"9df4016a-4b05-4252-8246-98e8c1e6cce4\") " Nov 03 21:32:18.338816 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.338784 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df4016a-4b05-4252-8246-98e8c1e6cce4-plugins-config-volume" pod "9df4016a-4b05-4252-8246-98e8c1e6cce4" (UID: "9df4016a-4b05-4252-8246-98e8c1e6cce4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:32:18.338957 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.338897 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9df4016a-4b05-4252-8246-98e8c1e6cce4-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:18.340551 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.340526 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df4016a-4b05-4252-8246-98e8c1e6cce4-kube-api-access-sn8tv" pod "9df4016a-4b05-4252-8246-98e8c1e6cce4" (UID: "9df4016a-4b05-4252-8246-98e8c1e6cce4"). InnerVolumeSpecName "kube-api-access-sn8tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:32:18.359516 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.359491 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a41883-5cbdf8bb47-m697s" Nov 03 21:32:18.439614 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.439576 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5-plugins-config-volume\") pod \"db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5\" (UID: \"db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5\") " Nov 03 21:32:18.439805 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.439656 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5-kube-api-access-wtq8s\" (UniqueName: \"kubernetes.io/projected/db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5-kube-api-access-wtq8s\") pod \"db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5\" (UID: \"db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5\") " Nov 03 21:32:18.439996 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.439971 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5-plugins-config-volume" pod "db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5" (UID: "db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:32:18.440201 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.440185 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn8tv\" (UniqueName: \"kubernetes.io/projected/9df4016a-4b05-4252-8246-98e8c1e6cce4-kube-api-access-sn8tv\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:18.440250 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.440203 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:18.441849 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.441830 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5-kube-api-access-wtq8s" pod "db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5" (UID: "db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5"). InnerVolumeSpecName "kube-api-access-wtq8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:32:18.541612 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.541572 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtq8s\" (UniqueName: \"kubernetes.io/projected/db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5-kube-api-access-wtq8s\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:18.717737 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.717705 5615 generic.go:356] "Generic (PLEG): container finished" podID="9df4016a-4b05-4252-8246-98e8c1e6cce4" containerID="3235f5d0965066e4eb508533d78d7ede9e6ab23e47cca352fe9fce885c746b64" exitCode=0 Nov 03 21:32:18.717927 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.717750 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a41883-5d94544d6b-snqv5" Nov 03 21:32:18.717927 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.717794 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a41883-5d94544d6b-snqv5" event={"ID":"9df4016a-4b05-4252-8246-98e8c1e6cce4","Type":"ContainerDied","Data":"3235f5d0965066e4eb508533d78d7ede9e6ab23e47cca352fe9fce885c746b64"} Nov 03 21:32:18.717927 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.717847 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a41883-5d94544d6b-snqv5" event={"ID":"9df4016a-4b05-4252-8246-98e8c1e6cce4","Type":"ContainerDied","Data":"919abad6b2a28504c1a377c61e450fdf6c61cae8a5d2c793912de00a5c3e078d"} Nov 03 21:32:18.717927 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.717875 5615 scope.go:122] "RemoveContainer" containerID="3235f5d0965066e4eb508533d78d7ede9e6ab23e47cca352fe9fce885c746b64" Nov 03 21:32:18.719198 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.719175 5615 generic.go:356] "Generic (PLEG): container finished" podID="db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5" containerID="4e4b3851b4a49f6b1d4f73845fa8eb366d86229ab0f14296b4f732b68bf783b4" exitCode=0 Nov 03 21:32:18.719312 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.719207 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a41883-5cbdf8bb47-m697s" event={"ID":"db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5","Type":"ContainerDied","Data":"4e4b3851b4a49f6b1d4f73845fa8eb366d86229ab0f14296b4f732b68bf783b4"} Nov 03 21:32:18.719312 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.719225 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a41883-5cbdf8bb47-m697s" Nov 03 21:32:18.719312 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.719262 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a41883-5cbdf8bb47-m697s" event={"ID":"db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5","Type":"ContainerDied","Data":"d466826df0684711f35156c64c915af014b76f9260633401722874d3d41037aa"} Nov 03 21:32:18.728534 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.728512 5615 scope.go:122] "RemoveContainer" containerID="3235f5d0965066e4eb508533d78d7ede9e6ab23e47cca352fe9fce885c746b64" Nov 03 21:32:18.728839 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:32:18.728816 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3235f5d0965066e4eb508533d78d7ede9e6ab23e47cca352fe9fce885c746b64\": container with ID starting with 3235f5d0965066e4eb508533d78d7ede9e6ab23e47cca352fe9fce885c746b64 not found: ID does not exist" containerID="3235f5d0965066e4eb508533d78d7ede9e6ab23e47cca352fe9fce885c746b64" Nov 03 21:32:18.728914 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.728849 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3235f5d0965066e4eb508533d78d7ede9e6ab23e47cca352fe9fce885c746b64"} err="failed to get container status \"3235f5d0965066e4eb508533d78d7ede9e6ab23e47cca352fe9fce885c746b64\": rpc error: code = NotFound desc = could not find container \"3235f5d0965066e4eb508533d78d7ede9e6ab23e47cca352fe9fce885c746b64\": container with ID starting with 3235f5d0965066e4eb508533d78d7ede9e6ab23e47cca352fe9fce885c746b64 not found: ID does not exist" Nov 03 21:32:18.728914 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.728885 5615 scope.go:122] "RemoveContainer" containerID="4e4b3851b4a49f6b1d4f73845fa8eb366d86229ab0f14296b4f732b68bf783b4" Nov 03 21:32:18.738059 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.738040 5615 scope.go:122] "RemoveContainer" containerID="4e4b3851b4a49f6b1d4f73845fa8eb366d86229ab0f14296b4f732b68bf783b4" Nov 03 21:32:18.738364 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:32:18.738341 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e4b3851b4a49f6b1d4f73845fa8eb366d86229ab0f14296b4f732b68bf783b4\": container with ID starting with 4e4b3851b4a49f6b1d4f73845fa8eb366d86229ab0f14296b4f732b68bf783b4 not found: ID does not exist" containerID="4e4b3851b4a49f6b1d4f73845fa8eb366d86229ab0f14296b4f732b68bf783b4" Nov 03 21:32:18.738411 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.738373 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e4b3851b4a49f6b1d4f73845fa8eb366d86229ab0f14296b4f732b68bf783b4"} err="failed to get container status \"4e4b3851b4a49f6b1d4f73845fa8eb366d86229ab0f14296b4f732b68bf783b4\": rpc error: code = NotFound desc = could not find container \"4e4b3851b4a49f6b1d4f73845fa8eb366d86229ab0f14296b4f732b68bf783b4\": container with ID starting with 4e4b3851b4a49f6b1d4f73845fa8eb366d86229ab0f14296b4f732b68bf783b4 not found: ID does not exist" Nov 03 21:32:18.874560 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.874477 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a41883-5cbdf8bb47-m697s"] Nov 03 21:32:18.876940 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.876911 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a41883-5cbdf8bb47-m697s"] Nov 03 21:32:18.979197 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.979161 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a41883-5d94544d6b-snqv5"] Nov 03 21:32:18.983842 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:18.983813 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a41883-5d94544d6b-snqv5"] Nov 03 21:32:19.652630 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:19.652588 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9df4016a-4b05-4252-8246-98e8c1e6cce4" path="/var/lib/kubelet/pods/9df4016a-4b05-4252-8246-98e8c1e6cce4/volumes" Nov 03 21:32:19.653195 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:19.653172 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5" path="/var/lib/kubelet/pods/db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5/volumes" Nov 03 21:32:24.857662 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:24.857620 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb"] Nov 03 21:32:24.858305 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:24.858289 5615 cpu_manager.go:414] "containerMap: removing container" podUID="db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5" containerName="payload-pre-processing" Nov 03 21:32:24.858369 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:24.858307 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5" containerName="payload-pre-processing" Nov 03 21:32:24.858369 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:24.858327 5615 cpu_manager.go:414] "containerMap: removing container" podUID="9df4016a-4b05-4252-8246-98e8c1e6cce4" containerName="payload-processing" Nov 03 21:32:24.858369 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:24.858332 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df4016a-4b05-4252-8246-98e8c1e6cce4" containerName="payload-processing" Nov 03 21:32:24.858466 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:24.858427 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="db7cb3a0-c93d-4b12-9bb3-333d2c4e0ff5" containerName="payload-pre-processing" Nov 03 21:32:24.858466 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:24.858437 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df4016a-4b05-4252-8246-98e8c1e6cce4" containerName="payload-processing" Nov 03 21:32:24.861045 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:24.861029 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb" Nov 03 21:32:24.913584 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:24.913450 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-a1c634-serving-cert\"" Nov 03 21:32:24.933337 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:24.933299 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9592ace4-6808-4b59-8bbc-58243dc78e12-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb\" (UID: \"9592ace4-6808-4b59-8bbc-58243dc78e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb" Nov 03 21:32:24.933587 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:24.933564 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9592ace4-6808-4b59-8bbc-58243dc78e12-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb\" (UID: \"9592ace4-6808-4b59-8bbc-58243dc78e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb" Nov 03 21:32:24.933657 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:24.933612 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9592ace4-6808-4b59-8bbc-58243dc78e12-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb\" (UID: \"9592ace4-6808-4b59-8bbc-58243dc78e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb" Nov 03 21:32:24.933725 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:24.933703 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knzqh\" (UniqueName: \"kubernetes.io/projected/9592ace4-6808-4b59-8bbc-58243dc78e12-kube-api-access-knzqh\") pod \"maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb\" (UID: \"9592ace4-6808-4b59-8bbc-58243dc78e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb" Nov 03 21:32:24.963275 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:24.963237 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb"] Nov 03 21:32:25.035161 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:25.035113 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9592ace4-6808-4b59-8bbc-58243dc78e12-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb\" (UID: \"9592ace4-6808-4b59-8bbc-58243dc78e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb" Nov 03 21:32:25.035353 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:25.035297 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9592ace4-6808-4b59-8bbc-58243dc78e12-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb\" (UID: \"9592ace4-6808-4b59-8bbc-58243dc78e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb" Nov 03 21:32:25.035353 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:25.035336 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9592ace4-6808-4b59-8bbc-58243dc78e12-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb\" (UID: \"9592ace4-6808-4b59-8bbc-58243dc78e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb" Nov 03 21:32:25.035448 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:25.035379 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knzqh\" (UniqueName: \"kubernetes.io/projected/9592ace4-6808-4b59-8bbc-58243dc78e12-kube-api-access-knzqh\") pod \"maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb\" (UID: \"9592ace4-6808-4b59-8bbc-58243dc78e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb" Nov 03 21:32:25.035509 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:32:25.035462 5615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-serving-cert: secret "maas-api-e2e-shared-b-w4-a1c634-serving-cert" not found Nov 03 21:32:25.035572 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:32:25.035558 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9592ace4-6808-4b59-8bbc-58243dc78e12-maas-api-tls podName:9592ace4-6808-4b59-8bbc-58243dc78e12 nodeName:}" failed. No retries permitted until 2026-11-03 21:32:25.535533623 +0000 UTC m=+2640.003312326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9592ace4-6808-4b59-8bbc-58243dc78e12-maas-api-tls") pod "maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb" (UID: "9592ace4-6808-4b59-8bbc-58243dc78e12") : secret "maas-api-e2e-shared-b-w4-a1c634-serving-cert" not found Nov 03 21:32:25.035891 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:25.035870 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9592ace4-6808-4b59-8bbc-58243dc78e12-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb\" (UID: \"9592ace4-6808-4b59-8bbc-58243dc78e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb" Nov 03 21:32:25.037872 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:25.037852 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9592ace4-6808-4b59-8bbc-58243dc78e12-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb\" (UID: \"9592ace4-6808-4b59-8bbc-58243dc78e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb" Nov 03 21:32:25.096379 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:25.096341 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knzqh\" (UniqueName: \"kubernetes.io/projected/9592ace4-6808-4b59-8bbc-58243dc78e12-kube-api-access-knzqh\") pod \"maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb\" (UID: \"9592ace4-6808-4b59-8bbc-58243dc78e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb" Nov 03 21:32:25.543163 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:25.543116 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9592ace4-6808-4b59-8bbc-58243dc78e12-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb\" (UID: \"9592ace4-6808-4b59-8bbc-58243dc78e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb" Nov 03 21:32:25.545726 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:25.545693 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9592ace4-6808-4b59-8bbc-58243dc78e12-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb\" (UID: \"9592ace4-6808-4b59-8bbc-58243dc78e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb" Nov 03 21:32:25.773773 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:25.773744 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb" Nov 03 21:32:26.145208 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:26.145176 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb"] Nov 03 21:32:26.146637 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:32:26.146608 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9592ace4_6808_4b59_8bbc_58243dc78e12.slice/crio-7113d5f3c5930224c385aa95ec7366fa17b1119e1065a3b4beea4df949875183 WatchSource:0}: Error finding container 7113d5f3c5930224c385aa95ec7366fa17b1119e1065a3b4beea4df949875183: Status 404 returned error can't find the container with id 7113d5f3c5930224c385aa95ec7366fa17b1119e1065a3b4beea4df949875183 Nov 03 21:32:26.780422 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:26.780385 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb" event={"ID":"9592ace4-6808-4b59-8bbc-58243dc78e12","Type":"ContainerStarted","Data":"7113d5f3c5930224c385aa95ec7366fa17b1119e1065a3b4beea4df949875183"} Nov 03 21:32:27.799612 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:27.799566 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb" event={"ID":"9592ace4-6808-4b59-8bbc-58243dc78e12","Type":"ContainerStarted","Data":"ed6e5af3628f4c2ce87875ed85474f9af2d7dea14519c27483f324652ea6124f"} Nov 03 21:32:27.800117 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:27.799771 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb" Nov 03 21:32:27.912607 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:27.912547 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb" podStartSLOduration=2.942228469 podStartE2EDuration="3.912533072s" totalImagesPullingTime="970.304603ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:32:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:32:27.911600244 +0000 UTC m=+2642.379378977" watchObservedRunningTime="2026-11-03 21:32:27.912533072 +0000 UTC m=+2642.380311794" Nov 03 21:32:33.811355 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:33.811322 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb" Nov 03 21:32:37.998970 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:37.998931 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7"] Nov 03 21:32:37.999483 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:37.999172 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7" podUID="976a1cc4-cd02-4d49-b1a1-c324af201326" containerName="maas-api" containerID="cri-o://41e1ca7874f6b5dba71edd638617b2508c4b5a028bf4f7dc26f0767451fcad1d" gracePeriod=30 Nov 03 21:32:38.328696 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:38.328665 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7" Nov 03 21:32:38.377165 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:38.377131 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/976a1cc4-cd02-4d49-b1a1-c324af201326-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/976a1cc4-cd02-4d49-b1a1-c324af201326-maas-api-tls\") pod \"976a1cc4-cd02-4d49-b1a1-c324af201326\" (UID: \"976a1cc4-cd02-4d49-b1a1-c324af201326\") " Nov 03 21:32:38.377333 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:38.377278 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/976a1cc4-cd02-4d49-b1a1-c324af201326-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/976a1cc4-cd02-4d49-b1a1-c324af201326-openshift-service-ca\") pod \"976a1cc4-cd02-4d49-b1a1-c324af201326\" (UID: \"976a1cc4-cd02-4d49-b1a1-c324af201326\") " Nov 03 21:32:38.377333 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:38.377325 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/976a1cc4-cd02-4d49-b1a1-c324af201326-metrics-certs\" (UniqueName: \"kubernetes.io/secret/976a1cc4-cd02-4d49-b1a1-c324af201326-metrics-certs\") pod \"976a1cc4-cd02-4d49-b1a1-c324af201326\" (UID: \"976a1cc4-cd02-4d49-b1a1-c324af201326\") " Nov 03 21:32:38.377407 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:38.377350 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/976a1cc4-cd02-4d49-b1a1-c324af201326-kube-api-access-zcfm6\" (UniqueName: \"kubernetes.io/projected/976a1cc4-cd02-4d49-b1a1-c324af201326-kube-api-access-zcfm6\") pod \"976a1cc4-cd02-4d49-b1a1-c324af201326\" (UID: \"976a1cc4-cd02-4d49-b1a1-c324af201326\") " Nov 03 21:32:38.377679 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:38.377646 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/976a1cc4-cd02-4d49-b1a1-c324af201326-openshift-service-ca" pod "976a1cc4-cd02-4d49-b1a1-c324af201326" (UID: "976a1cc4-cd02-4d49-b1a1-c324af201326"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:32:38.377853 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:38.377831 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/976a1cc4-cd02-4d49-b1a1-c324af201326-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:38.379499 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:38.379466 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/976a1cc4-cd02-4d49-b1a1-c324af201326-metrics-certs" pod "976a1cc4-cd02-4d49-b1a1-c324af201326" (UID: "976a1cc4-cd02-4d49-b1a1-c324af201326"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:32:38.379499 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:38.379488 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/976a1cc4-cd02-4d49-b1a1-c324af201326-maas-api-tls" pod "976a1cc4-cd02-4d49-b1a1-c324af201326" (UID: "976a1cc4-cd02-4d49-b1a1-c324af201326"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:32:38.379632 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:38.379559 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/976a1cc4-cd02-4d49-b1a1-c324af201326-kube-api-access-zcfm6" pod "976a1cc4-cd02-4d49-b1a1-c324af201326" (UID: "976a1cc4-cd02-4d49-b1a1-c324af201326"). InnerVolumeSpecName "kube-api-access-zcfm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:32:38.478787 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:38.478744 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/976a1cc4-cd02-4d49-b1a1-c324af201326-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:38.478787 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:38.478780 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcfm6\" (UniqueName: \"kubernetes.io/projected/976a1cc4-cd02-4d49-b1a1-c324af201326-kube-api-access-zcfm6\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:38.479061 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:38.478797 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/976a1cc4-cd02-4d49-b1a1-c324af201326-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:38.939114 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:38.923208 5615 generic.go:356] "Generic (PLEG): container finished" podID="976a1cc4-cd02-4d49-b1a1-c324af201326" containerID="41e1ca7874f6b5dba71edd638617b2508c4b5a028bf4f7dc26f0767451fcad1d" exitCode=0 Nov 03 21:32:38.939114 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:38.923277 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7" event={"ID":"976a1cc4-cd02-4d49-b1a1-c324af201326","Type":"ContainerDied","Data":"41e1ca7874f6b5dba71edd638617b2508c4b5a028bf4f7dc26f0767451fcad1d"} Nov 03 21:32:38.939114 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:38.923331 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7" event={"ID":"976a1cc4-cd02-4d49-b1a1-c324af201326","Type":"ContainerDied","Data":"8225a65ee5c823f27962b98cddbff40c215f5cdd1377068e42ec0cd43acca5f7"} Nov 03 21:32:38.939114 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:38.923361 5615 scope.go:122] "RemoveContainer" containerID="41e1ca7874f6b5dba71edd638617b2508c4b5a028bf4f7dc26f0767451fcad1d" Nov 03 21:32:38.939114 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:38.923526 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7" Nov 03 21:32:39.002568 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:39.001385 5615 scope.go:122] "RemoveContainer" containerID="41e1ca7874f6b5dba71edd638617b2508c4b5a028bf4f7dc26f0767451fcad1d" Nov 03 21:32:39.002568 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:32:39.001774 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41e1ca7874f6b5dba71edd638617b2508c4b5a028bf4f7dc26f0767451fcad1d\": container with ID starting with 41e1ca7874f6b5dba71edd638617b2508c4b5a028bf4f7dc26f0767451fcad1d not found: ID does not exist" containerID="41e1ca7874f6b5dba71edd638617b2508c4b5a028bf4f7dc26f0767451fcad1d" Nov 03 21:32:39.002568 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:39.001810 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41e1ca7874f6b5dba71edd638617b2508c4b5a028bf4f7dc26f0767451fcad1d"} err="failed to get container status \"41e1ca7874f6b5dba71edd638617b2508c4b5a028bf4f7dc26f0767451fcad1d\": rpc error: code = NotFound desc = could not find container \"41e1ca7874f6b5dba71edd638617b2508c4b5a028bf4f7dc26f0767451fcad1d\": container with ID starting with 41e1ca7874f6b5dba71edd638617b2508c4b5a028bf4f7dc26f0767451fcad1d not found: ID does not exist" Nov 03 21:32:39.124868 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:39.124833 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7"] Nov 03 21:32:39.128429 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:39.128398 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a41883-55dfb7bfcc-dhdz7"] Nov 03 21:32:39.444395 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:39.444357 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r"] Nov 03 21:32:39.445114 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:39.445055 5615 cpu_manager.go:414] "containerMap: removing container" podUID="976a1cc4-cd02-4d49-b1a1-c324af201326" containerName="maas-api" Nov 03 21:32:39.445114 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:39.445111 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="976a1cc4-cd02-4d49-b1a1-c324af201326" containerName="maas-api" Nov 03 21:32:39.445287 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:39.445236 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="976a1cc4-cd02-4d49-b1a1-c324af201326" containerName="maas-api" Nov 03 21:32:39.448957 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:39.448938 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r" Nov 03 21:32:39.500776 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:39.500748 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-a6219b-serving-cert\"" Nov 03 21:32:39.504517 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:39.504481 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r"] Nov 03 21:32:39.506622 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:39.506589 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/614c1da8-3cb5-4362-8774-b6b76ab79ac2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r\" (UID: \"614c1da8-3cb5-4362-8774-b6b76ab79ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r" Nov 03 21:32:39.506782 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:39.506664 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/614c1da8-3cb5-4362-8774-b6b76ab79ac2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r\" (UID: \"614c1da8-3cb5-4362-8774-b6b76ab79ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r" Nov 03 21:32:39.506916 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:39.506886 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/614c1da8-3cb5-4362-8774-b6b76ab79ac2-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r\" (UID: \"614c1da8-3cb5-4362-8774-b6b76ab79ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r" Nov 03 21:32:39.506983 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:39.506935 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k7r2\" (UniqueName: \"kubernetes.io/projected/614c1da8-3cb5-4362-8774-b6b76ab79ac2-kube-api-access-5k7r2\") pod \"maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r\" (UID: \"614c1da8-3cb5-4362-8774-b6b76ab79ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r" Nov 03 21:32:39.608693 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:39.608657 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/614c1da8-3cb5-4362-8774-b6b76ab79ac2-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r\" (UID: \"614c1da8-3cb5-4362-8774-b6b76ab79ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r" Nov 03 21:32:39.608867 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:39.608698 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k7r2\" (UniqueName: \"kubernetes.io/projected/614c1da8-3cb5-4362-8774-b6b76ab79ac2-kube-api-access-5k7r2\") pod \"maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r\" (UID: \"614c1da8-3cb5-4362-8774-b6b76ab79ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r" Nov 03 21:32:39.608867 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:39.608808 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/614c1da8-3cb5-4362-8774-b6b76ab79ac2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r\" (UID: \"614c1da8-3cb5-4362-8774-b6b76ab79ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r" Nov 03 21:32:39.608867 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:39.608842 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/614c1da8-3cb5-4362-8774-b6b76ab79ac2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r\" (UID: \"614c1da8-3cb5-4362-8774-b6b76ab79ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r" Nov 03 21:32:39.608971 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:32:39.608932 5615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-serving-cert: secret "maas-api-e2e-shared-b-w2-a6219b-serving-cert" not found Nov 03 21:32:39.609028 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:32:39.609006 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/614c1da8-3cb5-4362-8774-b6b76ab79ac2-maas-api-tls podName:614c1da8-3cb5-4362-8774-b6b76ab79ac2 nodeName:}" failed. No retries permitted until 2026-11-03 21:32:40.108984988 +0000 UTC m=+2654.576763704 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/614c1da8-3cb5-4362-8774-b6b76ab79ac2-maas-api-tls") pod "maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r" (UID: "614c1da8-3cb5-4362-8774-b6b76ab79ac2") : secret "maas-api-e2e-shared-b-w2-a6219b-serving-cert" not found Nov 03 21:32:39.609515 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:39.609492 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/614c1da8-3cb5-4362-8774-b6b76ab79ac2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r\" (UID: \"614c1da8-3cb5-4362-8774-b6b76ab79ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r" Nov 03 21:32:39.611203 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:39.611187 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/614c1da8-3cb5-4362-8774-b6b76ab79ac2-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r\" (UID: \"614c1da8-3cb5-4362-8774-b6b76ab79ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r" Nov 03 21:32:39.652247 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:39.652216 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="976a1cc4-cd02-4d49-b1a1-c324af201326" path="/var/lib/kubelet/pods/976a1cc4-cd02-4d49-b1a1-c324af201326/volumes" Nov 03 21:32:39.668143 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:39.668101 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k7r2\" (UniqueName: \"kubernetes.io/projected/614c1da8-3cb5-4362-8774-b6b76ab79ac2-kube-api-access-5k7r2\") pod \"maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r\" (UID: \"614c1da8-3cb5-4362-8774-b6b76ab79ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r" Nov 03 21:32:40.117203 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:40.117162 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/614c1da8-3cb5-4362-8774-b6b76ab79ac2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r\" (UID: \"614c1da8-3cb5-4362-8774-b6b76ab79ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r" Nov 03 21:32:40.119669 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:40.119638 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/614c1da8-3cb5-4362-8774-b6b76ab79ac2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r\" (UID: \"614c1da8-3cb5-4362-8774-b6b76ab79ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r" Nov 03 21:32:40.362925 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:40.362890 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r" Nov 03 21:32:40.621623 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:32:40.621584 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:32:40.736352 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:40.736317 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r"] Nov 03 21:32:40.737682 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:32:40.737643 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod614c1da8_3cb5_4362_8774_b6b76ab79ac2.slice/crio-cf319282cd8c6b1231351a10f200bc290b0166ebcb4a453817016408686c49f0 WatchSource:0}: Error finding container cf319282cd8c6b1231351a10f200bc290b0166ebcb4a453817016408686c49f0: Status 404 returned error can't find the container with id cf319282cd8c6b1231351a10f200bc290b0166ebcb4a453817016408686c49f0 Nov 03 21:32:40.940191 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:40.940149 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r" event={"ID":"614c1da8-3cb5-4362-8774-b6b76ab79ac2","Type":"ContainerStarted","Data":"cf319282cd8c6b1231351a10f200bc290b0166ebcb4a453817016408686c49f0"} Nov 03 21:32:41.948477 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:41.948375 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r" event={"ID":"614c1da8-3cb5-4362-8774-b6b76ab79ac2","Type":"ContainerStarted","Data":"36828cfd137c91db4b814b0520e8cb019965e5d2d6f3784157ded54a609bc2e6"} Nov 03 21:32:41.948477 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:41.948439 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r" Nov 03 21:32:42.060667 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:42.060613 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r" podStartSLOduration=2.107062421 podStartE2EDuration="3.060599758s" totalImagesPullingTime="953.537337ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:32:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:32:42.059841912 +0000 UTC m=+2656.527620635" watchObservedRunningTime="2026-11-03 21:32:42.060599758 +0000 UTC m=+2656.528378481" Nov 03 21:32:44.352770 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:44.352735 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc"] Nov 03 21:32:44.353256 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:44.352937 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" podUID="4c842d3a-c0e4-4703-acc7-77053dd626ba" containerName="istio-proxy" containerID="cri-o://656a2c2d142b7a5add4a0da2b96aa2decb24e56fbece2759e5c831699cb58cd5" gracePeriod=30 Nov 03 21:32:47.960634 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:47.960602 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r" Nov 03 21:32:49.033310 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.029924 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7"] Nov 03 21:32:49.037911 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.037881 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7" Nov 03 21:32:49.089118 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.089055 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7"] Nov 03 21:32:49.132973 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.132937 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-4d1799-serving-cert\"" Nov 03 21:32:49.159168 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.159131 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca66826-bea7-4c51-8fc0-b0e824ac5f2c-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7\" (UID: \"dca66826-bea7-4c51-8fc0-b0e824ac5f2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7" Nov 03 21:32:49.159378 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.159242 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca66826-bea7-4c51-8fc0-b0e824ac5f2c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7\" (UID: \"dca66826-bea7-4c51-8fc0-b0e824ac5f2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7" Nov 03 21:32:49.159378 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.159295 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njg54\" (UniqueName: \"kubernetes.io/projected/dca66826-bea7-4c51-8fc0-b0e824ac5f2c-kube-api-access-njg54\") pod \"maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7\" (UID: \"dca66826-bea7-4c51-8fc0-b0e824ac5f2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7" Nov 03 21:32:49.159522 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.159466 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca66826-bea7-4c51-8fc0-b0e824ac5f2c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7\" (UID: \"dca66826-bea7-4c51-8fc0-b0e824ac5f2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7" Nov 03 21:32:49.264097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.263470 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca66826-bea7-4c51-8fc0-b0e824ac5f2c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7\" (UID: \"dca66826-bea7-4c51-8fc0-b0e824ac5f2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7" Nov 03 21:32:49.264097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.263554 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njg54\" (UniqueName: \"kubernetes.io/projected/dca66826-bea7-4c51-8fc0-b0e824ac5f2c-kube-api-access-njg54\") pod \"maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7\" (UID: \"dca66826-bea7-4c51-8fc0-b0e824ac5f2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7" Nov 03 21:32:49.264097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.263693 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca66826-bea7-4c51-8fc0-b0e824ac5f2c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7\" (UID: \"dca66826-bea7-4c51-8fc0-b0e824ac5f2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7" Nov 03 21:32:49.264097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.263992 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca66826-bea7-4c51-8fc0-b0e824ac5f2c-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7\" (UID: \"dca66826-bea7-4c51-8fc0-b0e824ac5f2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7" Nov 03 21:32:49.275936 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.273015 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca66826-bea7-4c51-8fc0-b0e824ac5f2c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7\" (UID: \"dca66826-bea7-4c51-8fc0-b0e824ac5f2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7" Nov 03 21:32:49.282103 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.280758 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca66826-bea7-4c51-8fc0-b0e824ac5f2c-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7\" (UID: \"dca66826-bea7-4c51-8fc0-b0e824ac5f2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7" Nov 03 21:32:49.288655 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.288570 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca66826-bea7-4c51-8fc0-b0e824ac5f2c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7\" (UID: \"dca66826-bea7-4c51-8fc0-b0e824ac5f2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7" Nov 03 21:32:49.324269 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.324232 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njg54\" (UniqueName: \"kubernetes.io/projected/dca66826-bea7-4c51-8fc0-b0e824ac5f2c-kube-api-access-njg54\") pod \"maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7\" (UID: \"dca66826-bea7-4c51-8fc0-b0e824ac5f2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7" Nov 03 21:32:49.352234 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.352202 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7" Nov 03 21:32:49.633705 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.633678 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:32:49.771800 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.771756 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-workload-socket\") pod \"4c842d3a-c0e4-4703-acc7-77053dd626ba\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " Nov 03 21:32:49.771964 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.771817 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-istio-data\") pod \"4c842d3a-c0e4-4703-acc7-77053dd626ba\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " Nov 03 21:32:49.772070 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.772007 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c842d3a-c0e4-4703-acc7-77053dd626ba-kube-api-access-m4gnh\" (UniqueName: \"kubernetes.io/projected/4c842d3a-c0e4-4703-acc7-77053dd626ba-kube-api-access-m4gnh\") pod \"4c842d3a-c0e4-4703-acc7-77053dd626ba\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " Nov 03 21:32:49.772185 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.772151 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-istio-data" pod "4c842d3a-c0e4-4703-acc7-77053dd626ba" (UID: "4c842d3a-c0e4-4703-acc7-77053dd626ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:32:49.772252 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.772176 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-workload-socket" pod "4c842d3a-c0e4-4703-acc7-77053dd626ba" (UID: "4c842d3a-c0e4-4703-acc7-77053dd626ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:32:49.772775 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.772748 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c842d3a-c0e4-4703-acc7-77053dd626ba-istio-token\" (UniqueName: \"kubernetes.io/projected/4c842d3a-c0e4-4703-acc7-77053dd626ba-istio-token\") pod \"4c842d3a-c0e4-4703-acc7-77053dd626ba\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " Nov 03 21:32:49.772896 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.772808 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-workload-certs\") pod \"4c842d3a-c0e4-4703-acc7-77053dd626ba\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " Nov 03 21:32:49.772971 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.772909 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-istio-envoy\") pod \"4c842d3a-c0e4-4703-acc7-77053dd626ba\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " Nov 03 21:32:49.773063 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.773043 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c842d3a-c0e4-4703-acc7-77053dd626ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c842d3a-c0e4-4703-acc7-77053dd626ba-istio-podinfo\") pod \"4c842d3a-c0e4-4703-acc7-77053dd626ba\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " Nov 03 21:32:49.773135 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.773107 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c842d3a-c0e4-4703-acc7-77053dd626ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c842d3a-c0e4-4703-acc7-77053dd626ba-istiod-ca-cert\") pod \"4c842d3a-c0e4-4703-acc7-77053dd626ba\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " Nov 03 21:32:49.773203 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.773138 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-credential-socket\") pod \"4c842d3a-c0e4-4703-acc7-77053dd626ba\" (UID: \"4c842d3a-c0e4-4703-acc7-77053dd626ba\") " Nov 03 21:32:49.773203 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.773139 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-workload-certs" pod "4c842d3a-c0e4-4703-acc7-77053dd626ba" (UID: "4c842d3a-c0e4-4703-acc7-77053dd626ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:32:49.773651 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.773578 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-credential-socket" pod "4c842d3a-c0e4-4703-acc7-77053dd626ba" (UID: "4c842d3a-c0e4-4703-acc7-77053dd626ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:32:49.773651 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.773624 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c842d3a-c0e4-4703-acc7-77053dd626ba-istiod-ca-cert" pod "4c842d3a-c0e4-4703-acc7-77053dd626ba" (UID: "4c842d3a-c0e4-4703-acc7-77053dd626ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:32:49.773860 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.773836 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:49.773936 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.773865 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:49.773936 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.773880 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:49.773936 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.773896 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c842d3a-c0e4-4703-acc7-77053dd626ba-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:49.773936 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.773910 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:49.774631 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.774610 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c842d3a-c0e4-4703-acc7-77053dd626ba-kube-api-access-m4gnh" pod "4c842d3a-c0e4-4703-acc7-77053dd626ba" (UID: "4c842d3a-c0e4-4703-acc7-77053dd626ba"). InnerVolumeSpecName "kube-api-access-m4gnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:32:49.775266 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.775249 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c842d3a-c0e4-4703-acc7-77053dd626ba-istio-podinfo" pod "4c842d3a-c0e4-4703-acc7-77053dd626ba" (UID: "4c842d3a-c0e4-4703-acc7-77053dd626ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:32:49.775391 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.775377 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c842d3a-c0e4-4703-acc7-77053dd626ba-istio-token" pod "4c842d3a-c0e4-4703-acc7-77053dd626ba" (UID: "4c842d3a-c0e4-4703-acc7-77053dd626ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:32:49.776017 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.776003 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-istio-envoy" pod "4c842d3a-c0e4-4703-acc7-77053dd626ba" (UID: "4c842d3a-c0e4-4703-acc7-77053dd626ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:32:49.800438 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.800407 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7"] Nov 03 21:32:49.802053 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:32:49.802022 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca66826_bea7_4c51_8fc0_b0e824ac5f2c.slice/crio-08352a8179ff4ad1153c5a4c29bf24164d7415b7891cde6593f70fb1261d9eb6 WatchSource:0}: Error finding container 08352a8179ff4ad1153c5a4c29bf24164d7415b7891cde6593f70fb1261d9eb6: Status 404 returned error can't find the container with id 08352a8179ff4ad1153c5a4c29bf24164d7415b7891cde6593f70fb1261d9eb6 Nov 03 21:32:49.875766 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.875723 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c842d3a-c0e4-4703-acc7-77053dd626ba-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:49.875766 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.875755 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c842d3a-c0e4-4703-acc7-77053dd626ba-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:49.875766 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.875766 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c842d3a-c0e4-4703-acc7-77053dd626ba-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:49.876021 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:49.875775 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4gnh\" (UniqueName: \"kubernetes.io/projected/4c842d3a-c0e4-4703-acc7-77053dd626ba-kube-api-access-m4gnh\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:50.000503 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:50.000461 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7" event={"ID":"dca66826-bea7-4c51-8fc0-b0e824ac5f2c","Type":"ContainerStarted","Data":"08352a8179ff4ad1153c5a4c29bf24164d7415b7891cde6593f70fb1261d9eb6"} Nov 03 21:32:50.002223 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:50.002188 5615 generic.go:356] "Generic (PLEG): container finished" podID="4c842d3a-c0e4-4703-acc7-77053dd626ba" containerID="656a2c2d142b7a5add4a0da2b96aa2decb24e56fbece2759e5c831699cb58cd5" exitCode=0 Nov 03 21:32:50.002373 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:50.002252 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" Nov 03 21:32:50.002373 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:50.002273 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" event={"ID":"4c842d3a-c0e4-4703-acc7-77053dd626ba","Type":"ContainerDied","Data":"656a2c2d142b7a5add4a0da2b96aa2decb24e56fbece2759e5c831699cb58cd5"} Nov 03 21:32:50.002373 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:50.002337 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc" event={"ID":"4c842d3a-c0e4-4703-acc7-77053dd626ba","Type":"ContainerDied","Data":"770ac3cdc709f6f083d05f3ce8c478018b4f2cd2eadb309dbd1e4462de8566c5"} Nov 03 21:32:50.002373 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:50.002362 5615 scope.go:122] "RemoveContainer" containerID="656a2c2d142b7a5add4a0da2b96aa2decb24e56fbece2759e5c831699cb58cd5" Nov 03 21:32:50.013156 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:50.013136 5615 scope.go:122] "RemoveContainer" containerID="b6a0b200e52325e74816d5357db26ea82cfb519572aedac14ef6e19ea27b4e3d" Nov 03 21:32:50.023566 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:50.023541 5615 scope.go:122] "RemoveContainer" containerID="656a2c2d142b7a5add4a0da2b96aa2decb24e56fbece2759e5c831699cb58cd5" Nov 03 21:32:50.023900 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:32:50.023876 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"656a2c2d142b7a5add4a0da2b96aa2decb24e56fbece2759e5c831699cb58cd5\": container with ID starting with 656a2c2d142b7a5add4a0da2b96aa2decb24e56fbece2759e5c831699cb58cd5 not found: ID does not exist" containerID="656a2c2d142b7a5add4a0da2b96aa2decb24e56fbece2759e5c831699cb58cd5" Nov 03 21:32:50.023982 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:50.023910 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"656a2c2d142b7a5add4a0da2b96aa2decb24e56fbece2759e5c831699cb58cd5"} err="failed to get container status \"656a2c2d142b7a5add4a0da2b96aa2decb24e56fbece2759e5c831699cb58cd5\": rpc error: code = NotFound desc = could not find container \"656a2c2d142b7a5add4a0da2b96aa2decb24e56fbece2759e5c831699cb58cd5\": container with ID starting with 656a2c2d142b7a5add4a0da2b96aa2decb24e56fbece2759e5c831699cb58cd5 not found: ID does not exist" Nov 03 21:32:50.023982 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:50.023944 5615 scope.go:122] "RemoveContainer" containerID="b6a0b200e52325e74816d5357db26ea82cfb519572aedac14ef6e19ea27b4e3d" Nov 03 21:32:50.024252 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:32:50.024234 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6a0b200e52325e74816d5357db26ea82cfb519572aedac14ef6e19ea27b4e3d\": container with ID starting with b6a0b200e52325e74816d5357db26ea82cfb519572aedac14ef6e19ea27b4e3d not found: ID does not exist" containerID="b6a0b200e52325e74816d5357db26ea82cfb519572aedac14ef6e19ea27b4e3d" Nov 03 21:32:50.024303 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:50.024259 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6a0b200e52325e74816d5357db26ea82cfb519572aedac14ef6e19ea27b4e3d"} err="failed to get container status \"b6a0b200e52325e74816d5357db26ea82cfb519572aedac14ef6e19ea27b4e3d\": rpc error: code = NotFound desc = could not find container \"b6a0b200e52325e74816d5357db26ea82cfb519572aedac14ef6e19ea27b4e3d\": container with ID starting with b6a0b200e52325e74816d5357db26ea82cfb519572aedac14ef6e19ea27b4e3d not found: ID does not exist" Nov 03 21:32:50.159603 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:50.159517 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc"] Nov 03 21:32:50.163523 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:50.163493 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-a41883-openshift-default-69b69d4f85-lw5mc"] Nov 03 21:32:51.011037 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:51.010983 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7" event={"ID":"dca66826-bea7-4c51-8fc0-b0e824ac5f2c","Type":"ContainerStarted","Data":"f98b553f1c4b219392f6f24a0d21c6876f5cb0cafd6ed433a90eb56426463e89"} Nov 03 21:32:51.011232 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:51.011051 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7" Nov 03 21:32:51.135145 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:51.135086 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7" podStartSLOduration=2.244461492 podStartE2EDuration="3.135056636s" totalImagesPullingTime="890.595144ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:32:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:32:51.132852839 +0000 UTC m=+2665.600631564" watchObservedRunningTime="2026-11-03 21:32:51.135056636 +0000 UTC m=+2665.602835360" Nov 03 21:32:51.652764 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:51.652729 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c842d3a-c0e4-4703-acc7-77053dd626ba" path="/var/lib/kubelet/pods/4c842d3a-c0e4-4703-acc7-77053dd626ba/volumes" Nov 03 21:32:57.025170 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:57.025136 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7" Nov 03 21:32:57.816141 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:57.816104 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b"] Nov 03 21:32:57.816340 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:57.816308 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b" podUID="190b83fb-12d3-4fa6-97f8-bfdb76990fe8" containerName="maas-api" containerID="cri-o://f2535f4e07e36c4fb454d4f52a46ebb59285c1b5db912d6e0e20cf686fc1e5f2" gracePeriod=30 Nov 03 21:32:58.134918 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:58.134282 5615 generic.go:356] "Generic (PLEG): container finished" podID="190b83fb-12d3-4fa6-97f8-bfdb76990fe8" containerID="f2535f4e07e36c4fb454d4f52a46ebb59285c1b5db912d6e0e20cf686fc1e5f2" exitCode=0 Nov 03 21:32:58.134918 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:58.134431 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b" event={"ID":"190b83fb-12d3-4fa6-97f8-bfdb76990fe8","Type":"ContainerDied","Data":"f2535f4e07e36c4fb454d4f52a46ebb59285c1b5db912d6e0e20cf686fc1e5f2"} Nov 03 21:32:58.158720 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:58.158690 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b" Nov 03 21:32:58.234787 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:58.234752 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-metrics-certs\") pod \"190b83fb-12d3-4fa6-97f8-bfdb76990fe8\" (UID: \"190b83fb-12d3-4fa6-97f8-bfdb76990fe8\") " Nov 03 21:32:58.234968 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:58.234809 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-maas-api-tls\") pod \"190b83fb-12d3-4fa6-97f8-bfdb76990fe8\" (UID: \"190b83fb-12d3-4fa6-97f8-bfdb76990fe8\") " Nov 03 21:32:58.234968 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:58.234832 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-openshift-service-ca\") pod \"190b83fb-12d3-4fa6-97f8-bfdb76990fe8\" (UID: \"190b83fb-12d3-4fa6-97f8-bfdb76990fe8\") " Nov 03 21:32:58.234968 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:58.234945 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-kube-api-access-bq7bw\" (UniqueName: \"kubernetes.io/projected/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-kube-api-access-bq7bw\") pod \"190b83fb-12d3-4fa6-97f8-bfdb76990fe8\" (UID: \"190b83fb-12d3-4fa6-97f8-bfdb76990fe8\") " Nov 03 21:32:58.235388 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:58.235354 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-openshift-service-ca" pod "190b83fb-12d3-4fa6-97f8-bfdb76990fe8" (UID: "190b83fb-12d3-4fa6-97f8-bfdb76990fe8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:32:58.237642 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:58.237596 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-kube-api-access-bq7bw" pod "190b83fb-12d3-4fa6-97f8-bfdb76990fe8" (UID: "190b83fb-12d3-4fa6-97f8-bfdb76990fe8"). InnerVolumeSpecName "kube-api-access-bq7bw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:32:58.237642 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:58.237612 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-metrics-certs" pod "190b83fb-12d3-4fa6-97f8-bfdb76990fe8" (UID: "190b83fb-12d3-4fa6-97f8-bfdb76990fe8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:32:58.237910 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:58.237890 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-maas-api-tls" pod "190b83fb-12d3-4fa6-97f8-bfdb76990fe8" (UID: "190b83fb-12d3-4fa6-97f8-bfdb76990fe8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:32:58.248736 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:58.248697 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:58.349902 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:58.349859 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq7bw\" (UniqueName: \"kubernetes.io/projected/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-kube-api-access-bq7bw\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:58.349902 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:58.349902 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:58.349902 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:58.349911 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/190b83fb-12d3-4fa6-97f8-bfdb76990fe8-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:59.141758 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:59.141717 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b" event={"ID":"190b83fb-12d3-4fa6-97f8-bfdb76990fe8","Type":"ContainerDied","Data":"f8e79f853d68463951600f76e79334610aa5993ae8e3cefc37037d309fd9b9c5"} Nov 03 21:32:59.141758 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:59.141749 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b" Nov 03 21:32:59.142274 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:59.141786 5615 scope.go:122] "RemoveContainer" containerID="f2535f4e07e36c4fb454d4f52a46ebb59285c1b5db912d6e0e20cf686fc1e5f2" Nov 03 21:32:59.297860 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:59.297812 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b"] Nov 03 21:32:59.345319 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:59.345277 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc4de5-55b6db957f-d2k7b"] Nov 03 21:32:59.652590 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:32:59.652554 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="190b83fb-12d3-4fa6-97f8-bfdb76990fe8" path="/var/lib/kubelet/pods/190b83fb-12d3-4fa6-97f8-bfdb76990fe8/volumes" Nov 03 21:33:04.139421 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:04.139384 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp"] Nov 03 21:33:04.140010 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:04.139995 5615 cpu_manager.go:414] "containerMap: removing container" podUID="4c842d3a-c0e4-4703-acc7-77053dd626ba" containerName="istio-proxy" Nov 03 21:33:04.140053 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:04.140011 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c842d3a-c0e4-4703-acc7-77053dd626ba" containerName="istio-proxy" Nov 03 21:33:04.140053 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:04.140033 5615 cpu_manager.go:414] "containerMap: removing container" podUID="4c842d3a-c0e4-4703-acc7-77053dd626ba" containerName="istio-proxy" Nov 03 21:33:04.140053 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:04.140038 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c842d3a-c0e4-4703-acc7-77053dd626ba" containerName="istio-proxy" Nov 03 21:33:04.140053 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:04.140049 5615 cpu_manager.go:414] "containerMap: removing container" podUID="190b83fb-12d3-4fa6-97f8-bfdb76990fe8" containerName="maas-api" Nov 03 21:33:04.140214 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:04.140054 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="190b83fb-12d3-4fa6-97f8-bfdb76990fe8" containerName="maas-api" Nov 03 21:33:04.140214 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:04.140191 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c842d3a-c0e4-4703-acc7-77053dd626ba" containerName="istio-proxy" Nov 03 21:33:04.140214 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:04.140204 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="190b83fb-12d3-4fa6-97f8-bfdb76990fe8" containerName="maas-api" Nov 03 21:33:04.143501 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:04.143482 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp" Nov 03 21:33:04.243061 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:04.243026 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-f0dc66-serving-cert\"" Nov 03 21:33:04.247428 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:04.247403 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp"] Nov 03 21:33:04.265371 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:04.265336 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4271257f-77da-4bd1-9717-f7287bf6b713-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp\" (UID: \"4271257f-77da-4bd1-9717-f7287bf6b713\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp" Nov 03 21:33:04.265525 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:04.265465 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqrwz\" (UniqueName: \"kubernetes.io/projected/4271257f-77da-4bd1-9717-f7287bf6b713-kube-api-access-pqrwz\") pod \"maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp\" (UID: \"4271257f-77da-4bd1-9717-f7287bf6b713\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp" Nov 03 21:33:04.265525 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:04.265515 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4271257f-77da-4bd1-9717-f7287bf6b713-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp\" (UID: \"4271257f-77da-4bd1-9717-f7287bf6b713\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp" Nov 03 21:33:04.265740 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:04.265722 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4271257f-77da-4bd1-9717-f7287bf6b713-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp\" (UID: \"4271257f-77da-4bd1-9717-f7287bf6b713\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp" Nov 03 21:33:04.366692 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:04.366658 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4271257f-77da-4bd1-9717-f7287bf6b713-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp\" (UID: \"4271257f-77da-4bd1-9717-f7287bf6b713\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp" Nov 03 21:33:04.366869 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:04.366749 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqrwz\" (UniqueName: \"kubernetes.io/projected/4271257f-77da-4bd1-9717-f7287bf6b713-kube-api-access-pqrwz\") pod \"maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp\" (UID: \"4271257f-77da-4bd1-9717-f7287bf6b713\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp" Nov 03 21:33:04.366869 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:04.366795 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4271257f-77da-4bd1-9717-f7287bf6b713-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp\" (UID: \"4271257f-77da-4bd1-9717-f7287bf6b713\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp" Nov 03 21:33:04.366869 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:33:04.366820 5615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-serving-cert: secret "maas-api-e2e-shared-b-w0-f0dc66-serving-cert" not found Nov 03 21:33:04.366977 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:33:04.366897 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4271257f-77da-4bd1-9717-f7287bf6b713-maas-api-tls podName:4271257f-77da-4bd1-9717-f7287bf6b713 nodeName:}" failed. No retries permitted until 2026-11-03 21:33:04.866876436 +0000 UTC m=+2679.334655144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4271257f-77da-4bd1-9717-f7287bf6b713-maas-api-tls") pod "maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp" (UID: "4271257f-77da-4bd1-9717-f7287bf6b713") : secret "maas-api-e2e-shared-b-w0-f0dc66-serving-cert" not found Nov 03 21:33:04.367043 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:04.367023 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4271257f-77da-4bd1-9717-f7287bf6b713-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp\" (UID: \"4271257f-77da-4bd1-9717-f7287bf6b713\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp" Nov 03 21:33:04.367531 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:04.367512 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4271257f-77da-4bd1-9717-f7287bf6b713-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp\" (UID: \"4271257f-77da-4bd1-9717-f7287bf6b713\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp" Nov 03 21:33:04.369671 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:04.369644 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4271257f-77da-4bd1-9717-f7287bf6b713-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp\" (UID: \"4271257f-77da-4bd1-9717-f7287bf6b713\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp" Nov 03 21:33:04.449466 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:04.449383 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqrwz\" (UniqueName: \"kubernetes.io/projected/4271257f-77da-4bd1-9717-f7287bf6b713-kube-api-access-pqrwz\") pod \"maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp\" (UID: \"4271257f-77da-4bd1-9717-f7287bf6b713\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp" Nov 03 21:33:04.875137 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:04.875100 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4271257f-77da-4bd1-9717-f7287bf6b713-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp\" (UID: \"4271257f-77da-4bd1-9717-f7287bf6b713\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp" Nov 03 21:33:04.877704 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:04.877666 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4271257f-77da-4bd1-9717-f7287bf6b713-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp\" (UID: \"4271257f-77da-4bd1-9717-f7287bf6b713\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp" Nov 03 21:33:05.057559 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:05.057516 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp" Nov 03 21:33:05.452390 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:05.452357 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp"] Nov 03 21:33:05.454807 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:33:05.454778 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4271257f_77da_4bd1_9717_f7287bf6b713.slice/crio-ea6d4787616d82dee9dbcf174a8dc3564e4640bb29fdcbe939926f9ce1e2ab6f WatchSource:0}: Error finding container ea6d4787616d82dee9dbcf174a8dc3564e4640bb29fdcbe939926f9ce1e2ab6f: Status 404 returned error can't find the container with id ea6d4787616d82dee9dbcf174a8dc3564e4640bb29fdcbe939926f9ce1e2ab6f Nov 03 21:33:06.189962 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:06.189923 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp" event={"ID":"4271257f-77da-4bd1-9717-f7287bf6b713","Type":"ContainerStarted","Data":"ea6d4787616d82dee9dbcf174a8dc3564e4640bb29fdcbe939926f9ce1e2ab6f"} Nov 03 21:33:07.201411 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:07.201364 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp" event={"ID":"4271257f-77da-4bd1-9717-f7287bf6b713","Type":"ContainerStarted","Data":"b9d1d972590ab62d8739b3f193bf970fe7520dc4c7e69055f37252e1aeaf745f"} Nov 03 21:33:07.202564 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:07.202535 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp" Nov 03 21:33:10.337286 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:10.337244 5615 generic.go:356] "Generic (PLEG): container finished" podID="0fd70b58-6f75-415c-b26e-303faf172e6c" containerID="8b98f3786960a3457eddfefdc91186507ff2528407e8354152811066dd3f544e" exitCode=0 Nov 03 21:33:10.338069 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:10.337450 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" event={"ID":"0fd70b58-6f75-415c-b26e-303faf172e6c","Type":"ContainerDied","Data":"8b98f3786960a3457eddfefdc91186507ff2528407e8354152811066dd3f544e"} Nov 03 21:33:10.338673 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:10.338645 5615 scope.go:122] "RemoveContainer" containerID="8b98f3786960a3457eddfefdc91186507ff2528407e8354152811066dd3f544e" Nov 03 21:33:10.344576 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:10.344526 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:33:10.344708 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:10.344617 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:33:10.344708 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:10.344651 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:33:10.538897 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:10.538833 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp" podStartSLOduration=5.676681044 podStartE2EDuration="6.538817173s" totalImagesPullingTime="862.136129ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:33:07.31412426 +0000 UTC m=+2681.781902982" watchObservedRunningTime="2026-11-03 21:33:10.538817173 +0000 UTC m=+2685.006595963" Nov 03 21:33:10.626104 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:33:10.626001 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:33:10.749754 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:10.749720 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp"] Nov 03 21:33:10.749949 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:10.749928 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" podUID="618668eb-db73-4af8-967f-f73331c493f2" containerName="istio-proxy" containerID="cri-o://91b76ea1d242632b4fd940d3919d82fe541c0ce0074274bfc1905210e3c59e65" gracePeriod=30 Nov 03 21:33:11.347561 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:11.347526 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" event={"ID":"0fd70b58-6f75-415c-b26e-303faf172e6c","Type":"ContainerStarted","Data":"136f7114eef954168a0ef368c9a05ca5f7c7659623bb98a30aaa963bc13d1777"} Nov 03 21:33:11.985725 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:11.985689 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:33:12.326049 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:12.325952 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:33:12.327473 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:12.327445 5615 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 03 21:33:12.327606 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:12.327496 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" podUID="0fd70b58-6f75-415c-b26e-303faf172e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 03 21:33:13.326052 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:13.326017 5615 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 03 21:33:13.326548 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:13.326084 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" podUID="0fd70b58-6f75-415c-b26e-303faf172e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 03 21:33:14.271328 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:14.271290 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp" Nov 03 21:33:14.326263 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:14.326224 5615 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 03 21:33:14.326744 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:14.326285 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" podUID="0fd70b58-6f75-415c-b26e-303faf172e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 03 21:33:15.326623 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:15.326581 5615 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 03 21:33:15.327045 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:15.326643 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" podUID="0fd70b58-6f75-415c-b26e-303faf172e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 03 21:33:16.058586 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.058558 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:33:16.089596 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.089560 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/618668eb-db73-4af8-967f-f73331c493f2-kube-api-access-29m4x\" (UniqueName: \"kubernetes.io/projected/618668eb-db73-4af8-967f-f73331c493f2-kube-api-access-29m4x\") pod \"618668eb-db73-4af8-967f-f73331c493f2\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " Nov 03 21:33:16.089596 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.089598 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-istio-data\") pod \"618668eb-db73-4af8-967f-f73331c493f2\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " Nov 03 21:33:16.089846 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.089628 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/618668eb-db73-4af8-967f-f73331c493f2-istio-token\" (UniqueName: \"kubernetes.io/projected/618668eb-db73-4af8-967f-f73331c493f2-istio-token\") pod \"618668eb-db73-4af8-967f-f73331c493f2\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " Nov 03 21:33:16.089846 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.089690 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-istio-envoy\") pod \"618668eb-db73-4af8-967f-f73331c493f2\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " Nov 03 21:33:16.089846 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.089729 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-workload-socket\") pod \"618668eb-db73-4af8-967f-f73331c493f2\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " Nov 03 21:33:16.089846 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.089753 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/618668eb-db73-4af8-967f-f73331c493f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618668eb-db73-4af8-967f-f73331c493f2-istio-podinfo\") pod \"618668eb-db73-4af8-967f-f73331c493f2\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " Nov 03 21:33:16.090061 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.089975 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-istio-data" pod "618668eb-db73-4af8-967f-f73331c493f2" (UID: "618668eb-db73-4af8-967f-f73331c493f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:33:16.090130 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.090054 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-workload-socket" pod "618668eb-db73-4af8-967f-f73331c493f2" (UID: "618668eb-db73-4af8-967f-f73331c493f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:33:16.090193 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.090177 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-workload-certs\") pod \"618668eb-db73-4af8-967f-f73331c493f2\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " Nov 03 21:33:16.090250 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.090197 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/618668eb-db73-4af8-967f-f73331c493f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618668eb-db73-4af8-967f-f73331c493f2-istiod-ca-cert\") pod \"618668eb-db73-4af8-967f-f73331c493f2\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " Nov 03 21:33:16.090441 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.090412 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-workload-certs" pod "618668eb-db73-4af8-967f-f73331c493f2" (UID: "618668eb-db73-4af8-967f-f73331c493f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:33:16.090643 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.090591 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/618668eb-db73-4af8-967f-f73331c493f2-istiod-ca-cert" pod "618668eb-db73-4af8-967f-f73331c493f2" (UID: "618668eb-db73-4af8-967f-f73331c493f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:33:16.090808 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.090719 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-credential-socket" pod "618668eb-db73-4af8-967f-f73331c493f2" (UID: "618668eb-db73-4af8-967f-f73331c493f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:33:16.090808 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.090528 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-credential-socket\") pod \"618668eb-db73-4af8-967f-f73331c493f2\" (UID: \"618668eb-db73-4af8-967f-f73331c493f2\") " Nov 03 21:33:16.091337 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.091316 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:33:16.091386 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.091344 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:33:16.091386 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.091360 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618668eb-db73-4af8-967f-f73331c493f2-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:33:16.091386 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.091374 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:33:16.091476 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.091387 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:33:16.092303 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.092276 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/618668eb-db73-4af8-967f-f73331c493f2-istio-podinfo" pod "618668eb-db73-4af8-967f-f73331c493f2" (UID: "618668eb-db73-4af8-967f-f73331c493f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:33:16.092460 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.092404 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-istio-envoy" pod "618668eb-db73-4af8-967f-f73331c493f2" (UID: "618668eb-db73-4af8-967f-f73331c493f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:33:16.092460 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.092432 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/618668eb-db73-4af8-967f-f73331c493f2-istio-token" pod "618668eb-db73-4af8-967f-f73331c493f2" (UID: "618668eb-db73-4af8-967f-f73331c493f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:33:16.092562 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.092485 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/618668eb-db73-4af8-967f-f73331c493f2-kube-api-access-29m4x" pod "618668eb-db73-4af8-967f-f73331c493f2" (UID: "618668eb-db73-4af8-967f-f73331c493f2"). InnerVolumeSpecName "kube-api-access-29m4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:33:16.192612 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.192537 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29m4x\" (UniqueName: \"kubernetes.io/projected/618668eb-db73-4af8-967f-f73331c493f2-kube-api-access-29m4x\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:33:16.192612 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.192570 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618668eb-db73-4af8-967f-f73331c493f2-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:33:16.192612 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.192581 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618668eb-db73-4af8-967f-f73331c493f2-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:33:16.192612 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.192590 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618668eb-db73-4af8-967f-f73331c493f2-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:33:16.326718 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.326515 5615 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 03 21:33:16.326718 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.326569 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" podUID="0fd70b58-6f75-415c-b26e-303faf172e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 03 21:33:16.380037 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.379996 5615 generic.go:356] "Generic (PLEG): container finished" podID="618668eb-db73-4af8-967f-f73331c493f2" containerID="91b76ea1d242632b4fd940d3919d82fe541c0ce0074274bfc1905210e3c59e65" exitCode=0 Nov 03 21:33:16.380266 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.380050 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" Nov 03 21:33:16.380266 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.380097 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" event={"ID":"618668eb-db73-4af8-967f-f73331c493f2","Type":"ContainerDied","Data":"91b76ea1d242632b4fd940d3919d82fe541c0ce0074274bfc1905210e3c59e65"} Nov 03 21:33:16.380266 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.380158 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp" event={"ID":"618668eb-db73-4af8-967f-f73331c493f2","Type":"ContainerDied","Data":"70ea49a92b87c07828fdbc1cdad0dcd7b27abf7cb18145b41554dc74c548650a"} Nov 03 21:33:16.380266 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.380186 5615 scope.go:122] "RemoveContainer" containerID="91b76ea1d242632b4fd940d3919d82fe541c0ce0074274bfc1905210e3c59e65" Nov 03 21:33:16.391678 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.391654 5615 scope.go:122] "RemoveContainer" containerID="d18982066814a79e6da0455fccfcf4a09093d54b440c74083b43531b6929c5ed" Nov 03 21:33:16.423305 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.423281 5615 scope.go:122] "RemoveContainer" containerID="91b76ea1d242632b4fd940d3919d82fe541c0ce0074274bfc1905210e3c59e65" Nov 03 21:33:16.423697 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:33:16.423671 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91b76ea1d242632b4fd940d3919d82fe541c0ce0074274bfc1905210e3c59e65\": container with ID starting with 91b76ea1d242632b4fd940d3919d82fe541c0ce0074274bfc1905210e3c59e65 not found: ID does not exist" containerID="91b76ea1d242632b4fd940d3919d82fe541c0ce0074274bfc1905210e3c59e65" Nov 03 21:33:16.423768 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.423706 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91b76ea1d242632b4fd940d3919d82fe541c0ce0074274bfc1905210e3c59e65"} err="failed to get container status \"91b76ea1d242632b4fd940d3919d82fe541c0ce0074274bfc1905210e3c59e65\": rpc error: code = NotFound desc = could not find container \"91b76ea1d242632b4fd940d3919d82fe541c0ce0074274bfc1905210e3c59e65\": container with ID starting with 91b76ea1d242632b4fd940d3919d82fe541c0ce0074274bfc1905210e3c59e65 not found: ID does not exist" Nov 03 21:33:16.423768 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.423725 5615 scope.go:122] "RemoveContainer" containerID="d18982066814a79e6da0455fccfcf4a09093d54b440c74083b43531b6929c5ed" Nov 03 21:33:16.424002 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:33:16.423986 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d18982066814a79e6da0455fccfcf4a09093d54b440c74083b43531b6929c5ed\": container with ID starting with d18982066814a79e6da0455fccfcf4a09093d54b440c74083b43531b6929c5ed not found: ID does not exist" containerID="d18982066814a79e6da0455fccfcf4a09093d54b440c74083b43531b6929c5ed" Nov 03 21:33:16.424052 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.424009 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d18982066814a79e6da0455fccfcf4a09093d54b440c74083b43531b6929c5ed"} err="failed to get container status \"d18982066814a79e6da0455fccfcf4a09093d54b440c74083b43531b6929c5ed\": rpc error: code = NotFound desc = could not find container \"d18982066814a79e6da0455fccfcf4a09093d54b440c74083b43531b6929c5ed\": container with ID starting with d18982066814a79e6da0455fccfcf4a09093d54b440c74083b43531b6929c5ed not found: ID does not exist" Nov 03 21:33:16.538411 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.538377 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp"] Nov 03 21:33:16.541185 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:16.541155 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-cc4de5-openshift-default-7fccbfd88f-r76pp"] Nov 03 21:33:17.330533 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:17.330503 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:33:17.331571 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:17.331549 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 21:33:17.654888 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:17.654811 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="618668eb-db73-4af8-967f-f73331c493f2" path="/var/lib/kubelet/pods/618668eb-db73-4af8-967f-f73331c493f2/volumes" Nov 03 21:33:19.922926 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:19.922889 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb"] Nov 03 21:33:19.923657 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:19.923633 5615 cpu_manager.go:414] "containerMap: removing container" podUID="618668eb-db73-4af8-967f-f73331c493f2" containerName="istio-proxy" Nov 03 21:33:19.923791 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:19.923658 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="618668eb-db73-4af8-967f-f73331c493f2" containerName="istio-proxy" Nov 03 21:33:19.923791 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:19.923688 5615 cpu_manager.go:414] "containerMap: removing container" podUID="618668eb-db73-4af8-967f-f73331c493f2" containerName="istio-proxy" Nov 03 21:33:19.923791 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:19.923696 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="618668eb-db73-4af8-967f-f73331c493f2" containerName="istio-proxy" Nov 03 21:33:19.923955 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:19.923858 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c842d3a-c0e4-4703-acc7-77053dd626ba" containerName="istio-proxy" Nov 03 21:33:19.923955 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:19.923879 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="618668eb-db73-4af8-967f-f73331c493f2" containerName="istio-proxy" Nov 03 21:33:19.923955 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:19.923893 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="618668eb-db73-4af8-967f-f73331c493f2" containerName="istio-proxy" Nov 03 21:33:19.927874 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:19.927836 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:19.948852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:19.948811 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istio-data\") pod \"e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:19.949141 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:19.949115 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-workload-socket\") pod \"e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:19.949295 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:19.949277 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istio-envoy\") pod \"e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:19.949525 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:19.949451 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-credential-socket\") pod \"e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:19.950412 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:19.949933 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istiod-ca-cert\") pod \"e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:19.951182 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:19.951161 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-workload-certs\") pod \"e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:19.951329 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:19.951314 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5sbh\" (UniqueName: \"kubernetes.io/projected/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-kube-api-access-t5sbh\") pod \"e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:19.951477 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:19.951463 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istio-token\") pod \"e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:19.951718 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:19.951701 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istio-podinfo\") pod \"e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:19.990108 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:19.989939 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-b22936-openshift-default-dockercfg-6tdxv\"" Nov 03 21:33:20.057097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:20.056857 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istio-podinfo\") pod \"e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:20.057097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:20.056935 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istio-data\") pod \"e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:20.057097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:20.056988 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-workload-socket\") pod \"e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:20.057097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:20.057021 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istio-envoy\") pod \"e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:20.057488 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:20.057184 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-credential-socket\") pod \"e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:20.057488 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:20.057224 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istiod-ca-cert\") pod \"e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:20.057488 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:20.057302 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-workload-certs\") pod \"e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:20.057488 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:20.057350 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5sbh\" (UniqueName: \"kubernetes.io/projected/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-kube-api-access-t5sbh\") pod \"e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:20.057488 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:20.057408 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istio-token\") pod \"e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:20.106276 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:20.094967 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-workload-socket\") pod \"e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:20.125099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:20.109090 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istiod-ca-cert\") pod \"e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:20.125099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:20.109849 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-workload-certs\") pod \"e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:20.125099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:20.110114 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istio-data\") pod \"e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:20.125099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:20.110672 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-credential-socket\") pod \"e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:20.155875 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:20.155310 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb"] Nov 03 21:33:20.158805 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:20.158741 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istio-podinfo\") pod \"e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:20.162397 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:20.162270 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istio-envoy\") pod \"e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:20.173990 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:20.173902 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istio-token\") pod \"e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:20.220335 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:20.220187 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5sbh\" (UniqueName: \"kubernetes.io/projected/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-kube-api-access-t5sbh\") pod \"e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:20.293723 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:20.293691 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:20.759890 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:20.759858 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb"] Nov 03 21:33:20.761696 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:33:20.761669 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf6ed462_f4bd_4bfa_9085_a6dcb7b0003c.slice/crio-c5a0c4143b9762b1569e43500a4727506974437e734ad6a2e6fe65854db59a3b WatchSource:0}: Error finding container c5a0c4143b9762b1569e43500a4727506974437e734ad6a2e6fe65854db59a3b: Status 404 returned error can't find the container with id c5a0c4143b9762b1569e43500a4727506974437e734ad6a2e6fe65854db59a3b Nov 03 21:33:20.764542 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:20.764506 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:33:20.764602 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:20.764577 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:33:20.764644 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:20.764606 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:33:21.418412 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:21.418374 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" event={"ID":"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c","Type":"ContainerStarted","Data":"8b43aa028e5711c209c003852abc417446b84ec7da1205fd1260a477c808e5c3"} Nov 03 21:33:21.418858 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:21.418434 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" event={"ID":"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c","Type":"ContainerStarted","Data":"c5a0c4143b9762b1569e43500a4727506974437e734ad6a2e6fe65854db59a3b"} Nov 03 21:33:21.533738 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:21.533673 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" podStartSLOduration=2.533657667 podStartE2EDuration="2.533657667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:33:21.532580404 +0000 UTC m=+2696.000359126" watchObservedRunningTime="2026-11-03 21:33:21.533657667 +0000 UTC m=+2696.001436398" Nov 03 21:33:22.293987 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:22.293946 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:22.299879 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:22.299849 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:22.424561 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:22.424531 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:22.425489 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:22.425466 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:33:35.435299 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:35.435188 5615 patch_prober.go:29] interesting pod/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:33:35.435299 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:35.435255 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" podUID="ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:33:36.643356 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:36.643316 5615 generic.go:356] "Generic (PLEG): container finished" podID="ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829" containerID="7e3e4956bfdeeb65e4b0e46439f7a10fde5a99e478ff9f1e4216ff398956183e" exitCode=0 Nov 03 21:33:36.643799 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:36.643387 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" event={"ID":"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829","Type":"ContainerDied","Data":"7e3e4956bfdeeb65e4b0e46439f7a10fde5a99e478ff9f1e4216ff398956183e"} Nov 03 21:33:36.643998 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:36.643978 5615 scope.go:122] "RemoveContainer" containerID="7e3e4956bfdeeb65e4b0e46439f7a10fde5a99e478ff9f1e4216ff398956183e" Nov 03 21:33:36.646114 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:36.646060 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:33:36.646216 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:36.646175 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:33:36.646268 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:36.646227 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:33:37.653040 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:37.653000 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" event={"ID":"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829","Type":"ContainerStarted","Data":"6be1ac7260734d89285b74eedb028c32728be6f4bbe585e835863dc4c7d267e3"} Nov 03 21:33:37.787120 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:37.787038 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:33:37.787351 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:37.787325 5615 patch_prober.go:29] interesting pod/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 03 21:33:37.787447 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:37.787399 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" podUID="ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 03 21:33:38.787163 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:38.787127 5615 patch_prober.go:29] interesting pod/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 03 21:33:38.787583 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:38.787186 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" podUID="ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 03 21:33:39.787824 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:39.787783 5615 patch_prober.go:29] interesting pod/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 03 21:33:39.788287 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:39.787845 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" podUID="ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 03 21:33:40.644728 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:33:40.644689 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:33:40.787364 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:40.787323 5615 patch_prober.go:29] interesting pod/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 03 21:33:40.787561 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:40.787384 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" podUID="ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 03 21:33:41.787412 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:41.787377 5615 patch_prober.go:29] interesting pod/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 03 21:33:41.787855 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:41.787446 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" podUID="ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 03 21:33:42.787274 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:42.787213 5615 patch_prober.go:29] interesting pod/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 03 21:33:42.787500 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:42.787302 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" podUID="ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 03 21:33:43.791409 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:43.791377 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:33:43.791812 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:43.791775 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:33:43.792622 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:33:43.792600 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 21:34:00.768297 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:00.768258 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt"] Nov 03 21:34:00.771977 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:00.771946 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:00.988097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:00.988046 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-714979-openshift-default-dockercfg-z5v9v\"" Nov 03 21:34:01.003806 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:00.991979 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istio-data\") pod \"e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:01.003806 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:00.992090 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istio-envoy\") pod \"e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:01.003806 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:00.992139 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c78dk\" (UniqueName: \"kubernetes.io/projected/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-kube-api-access-c78dk\") pod \"e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:01.003806 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:00.992207 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-credential-socket\") pod \"e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:01.003806 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:00.992302 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istio-podinfo\") pod \"e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:01.003806 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:00.992349 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-workload-socket\") pod \"e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:01.003806 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:00.992403 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istiod-ca-cert\") pod \"e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:01.003806 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:00.992428 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istio-token\") pod \"e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:01.003806 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:00.992477 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-workload-certs\") pod \"e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:01.003806 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:00.998356 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt"] Nov 03 21:34:01.097810 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:01.094527 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istio-data\") pod \"e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:01.097810 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:01.094633 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istio-envoy\") pod \"e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:01.097810 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:01.094677 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c78dk\" (UniqueName: \"kubernetes.io/projected/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-kube-api-access-c78dk\") pod \"e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:01.097810 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:01.094744 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-credential-socket\") pod \"e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:01.097810 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:01.094841 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istio-podinfo\") pod \"e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:01.097810 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:01.094886 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-workload-socket\") pod \"e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:01.097810 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:01.094933 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istiod-ca-cert\") pod \"e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:01.097810 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:01.094958 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istio-token\") pod \"e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:01.097810 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:01.095003 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-workload-certs\") pod \"e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:01.097810 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:01.095035 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istio-data\") pod \"e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:01.117946 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:01.113648 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istio-podinfo\") pod \"e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:01.117946 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:01.113977 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-workload-socket\") pod \"e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:01.117946 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:01.114707 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istiod-ca-cert\") pod \"e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:01.117946 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:01.115293 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-workload-certs\") pod \"e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:01.117946 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:01.115820 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-credential-socket\") pod \"e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:01.120532 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:01.119634 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istio-envoy\") pod \"e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:01.173811 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:01.173771 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istio-token\") pod \"e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:01.174182 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:01.174150 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c78dk\" (UniqueName: \"kubernetes.io/projected/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-kube-api-access-c78dk\") pod \"e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:01.387372 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:01.387280 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:01.832299 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:01.832270 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt"] Nov 03 21:34:01.833820 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:34:01.833796 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad38eb17_97aa_4877_9efb_7e324f3ff5bd.slice/crio-aec92b4562f0b25c5915b5da6d767a1d14d33293c81ce6a065f5fa1ee1bbfaef WatchSource:0}: Error finding container aec92b4562f0b25c5915b5da6d767a1d14d33293c81ce6a065f5fa1ee1bbfaef: Status 404 returned error can't find the container with id aec92b4562f0b25c5915b5da6d767a1d14d33293c81ce6a065f5fa1ee1bbfaef Nov 03 21:34:01.836562 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:01.836528 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:34:01.836648 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:01.836595 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:34:01.836648 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:01.836625 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:34:02.039121 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:02.039068 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" event={"ID":"ad38eb17-97aa-4877-9efb-7e324f3ff5bd","Type":"ContainerStarted","Data":"f0f6f306addb0899d7820f1419b6b6424c128b0b68fc8f8a7e4b9784a6752d82"} Nov 03 21:34:02.039121 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:02.039137 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" event={"ID":"ad38eb17-97aa-4877-9efb-7e324f3ff5bd","Type":"ContainerStarted","Data":"aec92b4562f0b25c5915b5da6d767a1d14d33293c81ce6a065f5fa1ee1bbfaef"} Nov 03 21:34:02.158464 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:02.158348 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" podStartSLOduration=2.158328926 podStartE2EDuration="2.158328926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:34:02.156603507 +0000 UTC m=+2736.624382229" watchObservedRunningTime="2026-11-03 21:34:02.158328926 +0000 UTC m=+2736.626107652" Nov 03 21:34:02.388039 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:02.388005 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:02.393678 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:02.393651 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:03.045225 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:03.045195 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:03.046319 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:03.046299 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:34:10.620339 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:34:10.620297 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:34:40.619596 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:34:40.619555 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:34:43.436343 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:43.436300 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j"] Nov 03 21:34:43.440256 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:43.440233 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" Nov 03 21:34:43.538425 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:43.538371 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-9818d8\"/\"openshift-service-ca.crt\"" Nov 03 21:34:43.538425 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:43.538403 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-9818d8\"/\"default-dockercfg-74qdw\"" Nov 03 21:34:43.538701 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:43.538440 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-9818d8\"/\"kube-root-ca.crt\"" Nov 03 21:34:43.538701 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:43.538484 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-9818d8\"/\"auth-test-model-9818d8-kserve-self-signed-certs\"" Nov 03 21:34:43.541392 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:43.541361 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j"] Nov 03 21:34:43.646904 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:43.646146 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kljlg\" (UniqueName: \"kubernetes.io/projected/fd483f7c-f5d6-427f-a946-844ac94c9d9e-kube-api-access-kljlg\") pod \"auth-test-model-9818d8-kserve-5968dbccd9-cgn8j\" (UID: \"fd483f7c-f5d6-427f-a946-844ac94c9d9e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" Nov 03 21:34:43.647284 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:43.647255 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd483f7c-f5d6-427f-a946-844ac94c9d9e-tmp-dir\") pod \"auth-test-model-9818d8-kserve-5968dbccd9-cgn8j\" (UID: \"fd483f7c-f5d6-427f-a946-844ac94c9d9e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" Nov 03 21:34:43.647458 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:43.647430 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd483f7c-f5d6-427f-a946-844ac94c9d9e-tls-certs\") pod \"auth-test-model-9818d8-kserve-5968dbccd9-cgn8j\" (UID: \"fd483f7c-f5d6-427f-a946-844ac94c9d9e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" Nov 03 21:34:43.647591 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:43.647485 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd483f7c-f5d6-427f-a946-844ac94c9d9e-model-cache\") pod \"auth-test-model-9818d8-kserve-5968dbccd9-cgn8j\" (UID: \"fd483f7c-f5d6-427f-a946-844ac94c9d9e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" Nov 03 21:34:43.647591 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:43.647565 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd483f7c-f5d6-427f-a946-844ac94c9d9e-dshm\") pod \"auth-test-model-9818d8-kserve-5968dbccd9-cgn8j\" (UID: \"fd483f7c-f5d6-427f-a946-844ac94c9d9e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" Nov 03 21:34:43.647713 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:43.647677 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd483f7c-f5d6-427f-a946-844ac94c9d9e-home\") pod \"auth-test-model-9818d8-kserve-5968dbccd9-cgn8j\" (UID: \"fd483f7c-f5d6-427f-a946-844ac94c9d9e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" Nov 03 21:34:43.748738 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:43.748686 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kljlg\" (UniqueName: \"kubernetes.io/projected/fd483f7c-f5d6-427f-a946-844ac94c9d9e-kube-api-access-kljlg\") pod \"auth-test-model-9818d8-kserve-5968dbccd9-cgn8j\" (UID: \"fd483f7c-f5d6-427f-a946-844ac94c9d9e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" Nov 03 21:34:43.748924 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:43.748820 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd483f7c-f5d6-427f-a946-844ac94c9d9e-tmp-dir\") pod \"auth-test-model-9818d8-kserve-5968dbccd9-cgn8j\" (UID: \"fd483f7c-f5d6-427f-a946-844ac94c9d9e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" Nov 03 21:34:43.748924 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:43.748867 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd483f7c-f5d6-427f-a946-844ac94c9d9e-tls-certs\") pod \"auth-test-model-9818d8-kserve-5968dbccd9-cgn8j\" (UID: \"fd483f7c-f5d6-427f-a946-844ac94c9d9e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" Nov 03 21:34:43.748924 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:43.748904 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd483f7c-f5d6-427f-a946-844ac94c9d9e-model-cache\") pod \"auth-test-model-9818d8-kserve-5968dbccd9-cgn8j\" (UID: \"fd483f7c-f5d6-427f-a946-844ac94c9d9e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" Nov 03 21:34:43.749119 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:43.748945 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd483f7c-f5d6-427f-a946-844ac94c9d9e-dshm\") pod \"auth-test-model-9818d8-kserve-5968dbccd9-cgn8j\" (UID: \"fd483f7c-f5d6-427f-a946-844ac94c9d9e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" Nov 03 21:34:43.749119 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:43.749031 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd483f7c-f5d6-427f-a946-844ac94c9d9e-home\") pod \"auth-test-model-9818d8-kserve-5968dbccd9-cgn8j\" (UID: \"fd483f7c-f5d6-427f-a946-844ac94c9d9e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" Nov 03 21:34:43.749322 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:43.749297 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd483f7c-f5d6-427f-a946-844ac94c9d9e-tmp-dir\") pod \"auth-test-model-9818d8-kserve-5968dbccd9-cgn8j\" (UID: \"fd483f7c-f5d6-427f-a946-844ac94c9d9e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" Nov 03 21:34:43.749395 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:43.749347 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd483f7c-f5d6-427f-a946-844ac94c9d9e-model-cache\") pod \"auth-test-model-9818d8-kserve-5968dbccd9-cgn8j\" (UID: \"fd483f7c-f5d6-427f-a946-844ac94c9d9e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" Nov 03 21:34:43.749395 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:43.749378 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd483f7c-f5d6-427f-a946-844ac94c9d9e-home\") pod \"auth-test-model-9818d8-kserve-5968dbccd9-cgn8j\" (UID: \"fd483f7c-f5d6-427f-a946-844ac94c9d9e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" Nov 03 21:34:43.751441 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:43.751419 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd483f7c-f5d6-427f-a946-844ac94c9d9e-dshm\") pod \"auth-test-model-9818d8-kserve-5968dbccd9-cgn8j\" (UID: \"fd483f7c-f5d6-427f-a946-844ac94c9d9e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" Nov 03 21:34:43.751845 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:43.751825 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd483f7c-f5d6-427f-a946-844ac94c9d9e-tls-certs\") pod \"auth-test-model-9818d8-kserve-5968dbccd9-cgn8j\" (UID: \"fd483f7c-f5d6-427f-a946-844ac94c9d9e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" Nov 03 21:34:43.807120 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:43.807069 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kljlg\" (UniqueName: \"kubernetes.io/projected/fd483f7c-f5d6-427f-a946-844ac94c9d9e-kube-api-access-kljlg\") pod \"auth-test-model-9818d8-kserve-5968dbccd9-cgn8j\" (UID: \"fd483f7c-f5d6-427f-a946-844ac94c9d9e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" Nov 03 21:34:44.053518 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:44.053428 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" Nov 03 21:34:44.427490 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:44.427459 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j"] Nov 03 21:34:44.429346 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:34:44.429314 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd483f7c_f5d6_427f_a946_844ac94c9d9e.slice/crio-075316b55f4a75222bf7022ef51fb92826a7325d7aee49c7c122f7c2b5ff9e62 WatchSource:0}: Error finding container 075316b55f4a75222bf7022ef51fb92826a7325d7aee49c7c122f7c2b5ff9e62: Status 404 returned error can't find the container with id 075316b55f4a75222bf7022ef51fb92826a7325d7aee49c7c122f7c2b5ff9e62 Nov 03 21:34:45.357767 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:45.357722 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" event={"ID":"fd483f7c-f5d6-427f-a946-844ac94c9d9e","Type":"ContainerStarted","Data":"af48698100bc7efb6235af2692fb20dd7bd8c41cd90614c8014bb8c84ff8ce00"} Nov 03 21:34:45.357767 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:45.357771 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" event={"ID":"fd483f7c-f5d6-427f-a946-844ac94c9d9e","Type":"ContainerStarted","Data":"075316b55f4a75222bf7022ef51fb92826a7325d7aee49c7c122f7c2b5ff9e62"} Nov 03 21:34:45.470011 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:45.469954 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" podStartSLOduration=2.469938984 podStartE2EDuration="2.469938984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:34:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:34:45.46818357 +0000 UTC m=+2779.935962293" watchObservedRunningTime="2026-11-03 21:34:45.469938984 +0000 UTC m=+2779.937717707" Nov 03 21:34:46.054497 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:46.054466 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" Nov 03 21:34:54.054254 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:54.054217 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" Nov 03 21:34:54.067264 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:54.067236 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" Nov 03 21:34:54.449046 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:54.448962 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" Nov 03 21:34:58.477666 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:58.477626 5615 generic.go:356] "Generic (PLEG): container finished" podID="af6ed462-f4bd-4bfa-9085-a6dcb7b0003c" containerID="8b43aa028e5711c209c003852abc417446b84ec7da1205fd1260a477c808e5c3" exitCode=0 Nov 03 21:34:58.478233 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:58.477697 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" event={"ID":"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c","Type":"ContainerDied","Data":"8b43aa028e5711c209c003852abc417446b84ec7da1205fd1260a477c808e5c3"} Nov 03 21:34:58.478332 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:58.478314 5615 scope.go:122] "RemoveContainer" containerID="8b43aa028e5711c209c003852abc417446b84ec7da1205fd1260a477c808e5c3" Nov 03 21:34:58.480327 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:58.480292 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:34:58.480420 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:58.480389 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:34:58.480456 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:58.480440 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:34:59.486179 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:34:59.486142 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" event={"ID":"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c","Type":"ContainerStarted","Data":"3cb33d7d0c36b2821f42d997490d78543037d058db7e3f62a4bc37a874ae6b81"} Nov 03 21:35:00.294055 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:00.294021 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:35:00.294323 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:00.294296 5615 patch_prober.go:29] interesting pod/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 03 21:35:00.294386 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:00.294357 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" podUID="af6ed462-f4bd-4bfa-9085-a6dcb7b0003c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 03 21:35:01.294372 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:01.294334 5615 patch_prober.go:29] interesting pod/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 03 21:35:01.294801 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:01.294403 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" podUID="af6ed462-f4bd-4bfa-9085-a6dcb7b0003c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 03 21:35:02.294881 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:02.294839 5615 patch_prober.go:29] interesting pod/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 03 21:35:02.295336 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:02.294896 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" podUID="af6ed462-f4bd-4bfa-9085-a6dcb7b0003c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 03 21:35:03.294446 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:03.294412 5615 patch_prober.go:29] interesting pod/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 03 21:35:03.294636 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:03.294464 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" podUID="af6ed462-f4bd-4bfa-9085-a6dcb7b0003c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 03 21:35:04.294781 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:04.294741 5615 patch_prober.go:29] interesting pod/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 03 21:35:04.295224 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:04.294797 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" podUID="af6ed462-f4bd-4bfa-9085-a6dcb7b0003c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 03 21:35:05.299480 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:05.299447 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:35:05.299994 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:05.299782 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:35:05.300391 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:05.300370 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:35:10.638752 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:35:10.638667 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:35:40.620381 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:35:40.620338 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:35:41.833552 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:41.833511 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27"] Nov 03 21:35:41.834095 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:41.833774 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" podUID="1357b2a2-d12d-458e-942a-bdfc808972c1" containerName="istio-proxy" containerID="cri-o://936451a7b34040edff6140da36cd5624062df0e57dfaaec1a97c74213bad5b4c" gracePeriod=30 Nov 03 21:35:47.121733 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.121662 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:35:47.205658 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.205619 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-credential-socket\") pod \"1357b2a2-d12d-458e-942a-bdfc808972c1\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " Nov 03 21:35:47.205848 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.205739 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-istio-envoy\") pod \"1357b2a2-d12d-458e-942a-bdfc808972c1\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " Nov 03 21:35:47.205848 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.205769 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1357b2a2-d12d-458e-942a-bdfc808972c1-istio-token\" (UniqueName: \"kubernetes.io/projected/1357b2a2-d12d-458e-942a-bdfc808972c1-istio-token\") pod \"1357b2a2-d12d-458e-942a-bdfc808972c1\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " Nov 03 21:35:47.205848 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.205792 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1357b2a2-d12d-458e-942a-bdfc808972c1-kube-api-access-bz469\" (UniqueName: \"kubernetes.io/projected/1357b2a2-d12d-458e-942a-bdfc808972c1-kube-api-access-bz469\") pod \"1357b2a2-d12d-458e-942a-bdfc808972c1\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " Nov 03 21:35:47.206007 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.205846 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1357b2a2-d12d-458e-942a-bdfc808972c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1357b2a2-d12d-458e-942a-bdfc808972c1-istio-podinfo\") pod \"1357b2a2-d12d-458e-942a-bdfc808972c1\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " Nov 03 21:35:47.206007 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.205897 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-istio-data\") pod \"1357b2a2-d12d-458e-942a-bdfc808972c1\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " Nov 03 21:35:47.206007 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.205930 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-credential-socket" pod "1357b2a2-d12d-458e-942a-bdfc808972c1" (UID: "1357b2a2-d12d-458e-942a-bdfc808972c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:35:47.206200 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.206035 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-workload-certs\") pod \"1357b2a2-d12d-458e-942a-bdfc808972c1\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " Nov 03 21:35:47.206200 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.206089 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1357b2a2-d12d-458e-942a-bdfc808972c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1357b2a2-d12d-458e-942a-bdfc808972c1-istiod-ca-cert\") pod \"1357b2a2-d12d-458e-942a-bdfc808972c1\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " Nov 03 21:35:47.206320 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.206210 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-workload-socket\") pod \"1357b2a2-d12d-458e-942a-bdfc808972c1\" (UID: \"1357b2a2-d12d-458e-942a-bdfc808972c1\") " Nov 03 21:35:47.206320 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.206272 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-istio-data" pod "1357b2a2-d12d-458e-942a-bdfc808972c1" (UID: "1357b2a2-d12d-458e-942a-bdfc808972c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:35:47.206661 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.206550 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-workload-socket" pod "1357b2a2-d12d-458e-942a-bdfc808972c1" (UID: "1357b2a2-d12d-458e-942a-bdfc808972c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:35:47.206661 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.206642 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-workload-certs" pod "1357b2a2-d12d-458e-942a-bdfc808972c1" (UID: "1357b2a2-d12d-458e-942a-bdfc808972c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:35:47.206860 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.206605 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1357b2a2-d12d-458e-942a-bdfc808972c1-istiod-ca-cert" pod "1357b2a2-d12d-458e-942a-bdfc808972c1" (UID: "1357b2a2-d12d-458e-942a-bdfc808972c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:35:47.206860 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.206852 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:47.206978 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.206866 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1357b2a2-d12d-458e-942a-bdfc808972c1-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:47.206978 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.206876 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:47.206978 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.206884 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:47.206978 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.206893 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:47.208325 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.208301 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1357b2a2-d12d-458e-942a-bdfc808972c1-istio-token" pod "1357b2a2-d12d-458e-942a-bdfc808972c1" (UID: "1357b2a2-d12d-458e-942a-bdfc808972c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:35:47.208687 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.208660 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1357b2a2-d12d-458e-942a-bdfc808972c1-istio-podinfo" pod "1357b2a2-d12d-458e-942a-bdfc808972c1" (UID: "1357b2a2-d12d-458e-942a-bdfc808972c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:35:47.208687 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.208678 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1357b2a2-d12d-458e-942a-bdfc808972c1-kube-api-access-bz469" pod "1357b2a2-d12d-458e-942a-bdfc808972c1" (UID: "1357b2a2-d12d-458e-942a-bdfc808972c1"). InnerVolumeSpecName "kube-api-access-bz469". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:35:47.208917 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.208895 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-istio-envoy" pod "1357b2a2-d12d-458e-942a-bdfc808972c1" (UID: "1357b2a2-d12d-458e-942a-bdfc808972c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:35:47.308745 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.308711 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1357b2a2-d12d-458e-942a-bdfc808972c1-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:47.308745 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.308745 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1357b2a2-d12d-458e-942a-bdfc808972c1-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:47.308946 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.308755 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz469\" (UniqueName: \"kubernetes.io/projected/1357b2a2-d12d-458e-942a-bdfc808972c1-kube-api-access-bz469\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:47.308946 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.308767 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1357b2a2-d12d-458e-942a-bdfc808972c1-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:47.855531 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.855494 5615 generic.go:356] "Generic (PLEG): container finished" podID="1357b2a2-d12d-458e-942a-bdfc808972c1" containerID="936451a7b34040edff6140da36cd5624062df0e57dfaaec1a97c74213bad5b4c" exitCode=0 Nov 03 21:35:47.855724 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.855555 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" Nov 03 21:35:47.855724 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.855581 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" event={"ID":"1357b2a2-d12d-458e-942a-bdfc808972c1","Type":"ContainerDied","Data":"936451a7b34040edff6140da36cd5624062df0e57dfaaec1a97c74213bad5b4c"} Nov 03 21:35:47.855724 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.855654 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27" event={"ID":"1357b2a2-d12d-458e-942a-bdfc808972c1","Type":"ContainerDied","Data":"937f48c8b5378bd9db8591a10a30d8c87ee355f4bf45ee4105bccd7cee820a71"} Nov 03 21:35:47.855724 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.855689 5615 scope.go:122] "RemoveContainer" containerID="936451a7b34040edff6140da36cd5624062df0e57dfaaec1a97c74213bad5b4c" Nov 03 21:35:47.865982 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.865960 5615 scope.go:122] "RemoveContainer" containerID="936451a7b34040edff6140da36cd5624062df0e57dfaaec1a97c74213bad5b4c" Nov 03 21:35:47.866328 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:35:47.866310 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"936451a7b34040edff6140da36cd5624062df0e57dfaaec1a97c74213bad5b4c\": container with ID starting with 936451a7b34040edff6140da36cd5624062df0e57dfaaec1a97c74213bad5b4c not found: ID does not exist" containerID="936451a7b34040edff6140da36cd5624062df0e57dfaaec1a97c74213bad5b4c" Nov 03 21:35:47.866400 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:47.866339 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"936451a7b34040edff6140da36cd5624062df0e57dfaaec1a97c74213bad5b4c"} err="failed to get container status \"936451a7b34040edff6140da36cd5624062df0e57dfaaec1a97c74213bad5b4c\": rpc error: code = NotFound desc = could not find container \"936451a7b34040edff6140da36cd5624062df0e57dfaaec1a97c74213bad5b4c\": container with ID starting with 936451a7b34040edff6140da36cd5624062df0e57dfaaec1a97c74213bad5b4c not found: ID does not exist" Nov 03 21:35:48.061803 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:48.061762 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27"] Nov 03 21:35:48.110746 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:48.110698 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bwh27"] Nov 03 21:35:49.088215 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:49.088179 5615 patch_prober.go:29] interesting pod/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:35:49.088722 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:49.088245 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" podUID="ad38eb17-97aa-4877-9efb-7e324f3ff5bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:35:49.652795 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:49.652753 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1357b2a2-d12d-458e-942a-bdfc808972c1" path="/var/lib/kubelet/pods/1357b2a2-d12d-458e-942a-bdfc808972c1/volumes" Nov 03 21:35:50.879475 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:50.879438 5615 generic.go:356] "Generic (PLEG): container finished" podID="ad38eb17-97aa-4877-9efb-7e324f3ff5bd" containerID="f0f6f306addb0899d7820f1419b6b6424c128b0b68fc8f8a7e4b9784a6752d82" exitCode=0 Nov 03 21:35:50.879926 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:50.879508 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" event={"ID":"ad38eb17-97aa-4877-9efb-7e324f3ff5bd","Type":"ContainerDied","Data":"f0f6f306addb0899d7820f1419b6b6424c128b0b68fc8f8a7e4b9784a6752d82"} Nov 03 21:35:50.879993 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:50.879975 5615 scope.go:122] "RemoveContainer" containerID="f0f6f306addb0899d7820f1419b6b6424c128b0b68fc8f8a7e4b9784a6752d82" Nov 03 21:35:50.881893 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:50.881858 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:35:50.881993 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:50.881935 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:35:50.881993 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:50.881972 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:35:51.887015 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:51.886978 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" event={"ID":"ad38eb17-97aa-4877-9efb-7e324f3ff5bd","Type":"ContainerStarted","Data":"ce6ea373bc6574b8e309d0e99d458bee391a930bd8951b50a79b709a99b30c66"} Nov 03 21:35:52.388045 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:52.387997 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:35:52.388345 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:52.388322 5615 patch_prober.go:29] interesting pod/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 03 21:35:52.388405 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:52.388384 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" podUID="ad38eb17-97aa-4877-9efb-7e324f3ff5bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 03 21:35:53.388307 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:53.388268 5615 patch_prober.go:29] interesting pod/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 03 21:35:53.388741 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:53.388341 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" podUID="ad38eb17-97aa-4877-9efb-7e324f3ff5bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 03 21:35:54.388258 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:54.388217 5615 patch_prober.go:29] interesting pod/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 03 21:35:54.388445 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:54.388276 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" podUID="ad38eb17-97aa-4877-9efb-7e324f3ff5bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 03 21:35:55.388661 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:55.388626 5615 patch_prober.go:29] interesting pod/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 03 21:35:55.389124 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:55.388678 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" podUID="ad38eb17-97aa-4877-9efb-7e324f3ff5bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 03 21:35:56.387808 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:56.387775 5615 patch_prober.go:29] interesting pod/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 03 21:35:56.387974 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:56.387837 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" podUID="ad38eb17-97aa-4877-9efb-7e324f3ff5bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 03 21:35:57.392706 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:57.392672 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:35:57.393143 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:57.392955 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:35:57.393661 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:35:57.393643 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 21:36:00.238531 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:00.238494 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc"] Nov 03 21:36:00.239047 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:00.238759 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" podUID="7242ebfd-977b-4b38-9c9f-f9c125ef3280" containerName="istio-proxy" containerID="cri-o://ee5e48254a7196b196918bd8f446da4000f995ac12feadbb3630173d0e3a1c6b" gracePeriod=30 Nov 03 21:36:05.515333 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:05.515268 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:36:05.583010 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:05.582977 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7242ebfd-977b-4b38-9c9f-f9c125ef3280-kube-api-access-gfw49\" (UniqueName: \"kubernetes.io/projected/7242ebfd-977b-4b38-9c9f-f9c125ef3280-kube-api-access-gfw49\") pod \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " Nov 03 21:36:05.583231 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:05.583143 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-workload-socket\") pod \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " Nov 03 21:36:05.583231 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:05.583171 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istio-podinfo\") pod \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " Nov 03 21:36:05.583334 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:05.583249 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istio-token\" (UniqueName: \"kubernetes.io/projected/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istio-token\") pod \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " Nov 03 21:36:05.583334 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:05.583307 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istio-envoy\") pod \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " Nov 03 21:36:05.583434 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:05.583354 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istiod-ca-cert\") pod \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " Nov 03 21:36:05.583434 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:05.583412 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-credential-socket\") pod \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " Nov 03 21:36:05.583943 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:05.583439 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istio-data\") pod \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " Nov 03 21:36:05.583943 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:05.583463 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-workload-certs\") pod \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\" (UID: \"7242ebfd-977b-4b38-9c9f-f9c125ef3280\") " Nov 03 21:36:05.583943 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:05.583500 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-workload-socket" pod "7242ebfd-977b-4b38-9c9f-f9c125ef3280" (UID: "7242ebfd-977b-4b38-9c9f-f9c125ef3280"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:36:05.583943 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:05.583712 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-credential-socket" pod "7242ebfd-977b-4b38-9c9f-f9c125ef3280" (UID: "7242ebfd-977b-4b38-9c9f-f9c125ef3280"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:36:05.583943 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:05.583876 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istio-data" pod "7242ebfd-977b-4b38-9c9f-f9c125ef3280" (UID: "7242ebfd-977b-4b38-9c9f-f9c125ef3280"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:36:05.583943 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:05.583904 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-workload-certs" pod "7242ebfd-977b-4b38-9c9f-f9c125ef3280" (UID: "7242ebfd-977b-4b38-9c9f-f9c125ef3280"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:36:05.584454 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:05.584185 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:05.584454 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:05.584221 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:05.584454 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:05.584237 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:05.584454 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:05.584251 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:05.584454 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:05.584422 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istiod-ca-cert" pod "7242ebfd-977b-4b38-9c9f-f9c125ef3280" (UID: "7242ebfd-977b-4b38-9c9f-f9c125ef3280"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:36:05.585657 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:05.585635 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istio-envoy" pod "7242ebfd-977b-4b38-9c9f-f9c125ef3280" (UID: "7242ebfd-977b-4b38-9c9f-f9c125ef3280"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:36:05.585776 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:05.585760 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istio-podinfo" pod "7242ebfd-977b-4b38-9c9f-f9c125ef3280" (UID: "7242ebfd-977b-4b38-9c9f-f9c125ef3280"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:36:05.585858 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:05.585844 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istio-token" pod "7242ebfd-977b-4b38-9c9f-f9c125ef3280" (UID: "7242ebfd-977b-4b38-9c9f-f9c125ef3280"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:36:05.585923 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:05.585903 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7242ebfd-977b-4b38-9c9f-f9c125ef3280-kube-api-access-gfw49" pod "7242ebfd-977b-4b38-9c9f-f9c125ef3280" (UID: "7242ebfd-977b-4b38-9c9f-f9c125ef3280"). InnerVolumeSpecName "kube-api-access-gfw49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:36:05.686777 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:05.686742 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfw49\" (UniqueName: \"kubernetes.io/projected/7242ebfd-977b-4b38-9c9f-f9c125ef3280-kube-api-access-gfw49\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:05.687029 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:05.687011 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:05.687160 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:05.687034 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:05.687160 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:05.687048 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:05.687160 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:05.687061 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7242ebfd-977b-4b38-9c9f-f9c125ef3280-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:06.000283 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:06.000246 5615 generic.go:356] "Generic (PLEG): container finished" podID="7242ebfd-977b-4b38-9c9f-f9c125ef3280" containerID="ee5e48254a7196b196918bd8f446da4000f995ac12feadbb3630173d0e3a1c6b" exitCode=0 Nov 03 21:36:06.000493 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:06.000308 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" Nov 03 21:36:06.000493 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:06.000311 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" event={"ID":"7242ebfd-977b-4b38-9c9f-f9c125ef3280","Type":"ContainerDied","Data":"ee5e48254a7196b196918bd8f446da4000f995ac12feadbb3630173d0e3a1c6b"} Nov 03 21:36:06.000493 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:06.000439 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc" event={"ID":"7242ebfd-977b-4b38-9c9f-f9c125ef3280","Type":"ContainerDied","Data":"71fb4fabb2c3244f22f9fbdca1bf5a26500216ce0f145fe47a705623948c4460"} Nov 03 21:36:06.000493 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:06.000470 5615 scope.go:122] "RemoveContainer" containerID="ee5e48254a7196b196918bd8f446da4000f995ac12feadbb3630173d0e3a1c6b" Nov 03 21:36:06.010760 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:06.010734 5615 scope.go:122] "RemoveContainer" containerID="ee5e48254a7196b196918bd8f446da4000f995ac12feadbb3630173d0e3a1c6b" Nov 03 21:36:06.011034 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:36:06.011012 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee5e48254a7196b196918bd8f446da4000f995ac12feadbb3630173d0e3a1c6b\": container with ID starting with ee5e48254a7196b196918bd8f446da4000f995ac12feadbb3630173d0e3a1c6b not found: ID does not exist" containerID="ee5e48254a7196b196918bd8f446da4000f995ac12feadbb3630173d0e3a1c6b" Nov 03 21:36:06.011130 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:06.011043 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee5e48254a7196b196918bd8f446da4000f995ac12feadbb3630173d0e3a1c6b"} err="failed to get container status \"ee5e48254a7196b196918bd8f446da4000f995ac12feadbb3630173d0e3a1c6b\": rpc error: code = NotFound desc = could not find container \"ee5e48254a7196b196918bd8f446da4000f995ac12feadbb3630173d0e3a1c6b\": container with ID starting with ee5e48254a7196b196918bd8f446da4000f995ac12feadbb3630173d0e3a1c6b not found: ID does not exist" Nov 03 21:36:06.158760 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:06.158725 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc"] Nov 03 21:36:06.206168 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:06.206130 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-sz7fc"] Nov 03 21:36:07.653590 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:36:07.653550 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7242ebfd-977b-4b38-9c9f-f9c125ef3280" path="/var/lib/kubelet/pods/7242ebfd-977b-4b38-9c9f-f9c125ef3280/volumes" Nov 03 21:36:10.619436 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:36:10.619396 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:36:40.620362 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:36:40.620270 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:37:10.620087 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:37:10.620032 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:37:27.749043 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:27.748875 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf"] Nov 03 21:37:27.750027 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:27.749992 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" podUID="4b357d8d-9624-46e8-86bf-d046d06fee50" containerName="istio-proxy" containerID="cri-o://50cfe6497a22759d3f52657b444efc7b0be4e5bfc9d92ad0bf6d01dd07b72126" gracePeriod=30 Nov 03 21:37:33.023516 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.023485 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:37:33.161397 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.161313 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-credential-socket\") pod \"4b357d8d-9624-46e8-86bf-d046d06fee50\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " Nov 03 21:37:33.161397 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.161358 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b357d8d-9624-46e8-86bf-d046d06fee50-kube-api-access-nq79k\" (UniqueName: \"kubernetes.io/projected/4b357d8d-9624-46e8-86bf-d046d06fee50-kube-api-access-nq79k\") pod \"4b357d8d-9624-46e8-86bf-d046d06fee50\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " Nov 03 21:37:33.161591 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.161402 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b357d8d-9624-46e8-86bf-d046d06fee50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b357d8d-9624-46e8-86bf-d046d06fee50-istiod-ca-cert\") pod \"4b357d8d-9624-46e8-86bf-d046d06fee50\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " Nov 03 21:37:33.161591 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.161505 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-istio-envoy\") pod \"4b357d8d-9624-46e8-86bf-d046d06fee50\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " Nov 03 21:37:33.161660 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.161582 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-credential-socket" pod "4b357d8d-9624-46e8-86bf-d046d06fee50" (UID: "4b357d8d-9624-46e8-86bf-d046d06fee50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:37:33.161660 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.161586 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-istio-data\") pod \"4b357d8d-9624-46e8-86bf-d046d06fee50\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " Nov 03 21:37:33.161727 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.161669 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b357d8d-9624-46e8-86bf-d046d06fee50-istio-token\" (UniqueName: \"kubernetes.io/projected/4b357d8d-9624-46e8-86bf-d046d06fee50-istio-token\") pod \"4b357d8d-9624-46e8-86bf-d046d06fee50\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " Nov 03 21:37:33.161807 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.161782 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-workload-socket\") pod \"4b357d8d-9624-46e8-86bf-d046d06fee50\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " Nov 03 21:37:33.161807 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.161796 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-istio-data" pod "4b357d8d-9624-46e8-86bf-d046d06fee50" (UID: "4b357d8d-9624-46e8-86bf-d046d06fee50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:37:33.161933 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.161884 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b357d8d-9624-46e8-86bf-d046d06fee50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b357d8d-9624-46e8-86bf-d046d06fee50-istio-podinfo\") pod \"4b357d8d-9624-46e8-86bf-d046d06fee50\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " Nov 03 21:37:33.161933 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.161894 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b357d8d-9624-46e8-86bf-d046d06fee50-istiod-ca-cert" pod "4b357d8d-9624-46e8-86bf-d046d06fee50" (UID: "4b357d8d-9624-46e8-86bf-d046d06fee50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:37:33.162037 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.161936 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-workload-certs\") pod \"4b357d8d-9624-46e8-86bf-d046d06fee50\" (UID: \"4b357d8d-9624-46e8-86bf-d046d06fee50\") " Nov 03 21:37:33.162153 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.162127 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-workload-socket" pod "4b357d8d-9624-46e8-86bf-d046d06fee50" (UID: "4b357d8d-9624-46e8-86bf-d046d06fee50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:37:33.162341 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.162311 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-workload-certs" pod "4b357d8d-9624-46e8-86bf-d046d06fee50" (UID: "4b357d8d-9624-46e8-86bf-d046d06fee50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:37:33.162636 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.162618 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:37:33.162718 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.162645 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:37:33.162718 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.162659 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:37:33.162718 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.162673 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b357d8d-9624-46e8-86bf-d046d06fee50-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:37:33.162718 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.162686 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:37:33.163830 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.163794 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b357d8d-9624-46e8-86bf-d046d06fee50-kube-api-access-nq79k" pod "4b357d8d-9624-46e8-86bf-d046d06fee50" (UID: "4b357d8d-9624-46e8-86bf-d046d06fee50"). InnerVolumeSpecName "kube-api-access-nq79k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:37:33.163926 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.163875 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b357d8d-9624-46e8-86bf-d046d06fee50-istio-token" pod "4b357d8d-9624-46e8-86bf-d046d06fee50" (UID: "4b357d8d-9624-46e8-86bf-d046d06fee50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:37:33.164143 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.164122 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-istio-envoy" pod "4b357d8d-9624-46e8-86bf-d046d06fee50" (UID: "4b357d8d-9624-46e8-86bf-d046d06fee50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:37:33.164221 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.164208 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b357d8d-9624-46e8-86bf-d046d06fee50-istio-podinfo" pod "4b357d8d-9624-46e8-86bf-d046d06fee50" (UID: "4b357d8d-9624-46e8-86bf-d046d06fee50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:37:33.263723 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.263685 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b357d8d-9624-46e8-86bf-d046d06fee50-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:37:33.263723 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.263718 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq79k\" (UniqueName: \"kubernetes.io/projected/4b357d8d-9624-46e8-86bf-d046d06fee50-kube-api-access-nq79k\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:37:33.263723 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.263731 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b357d8d-9624-46e8-86bf-d046d06fee50-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:37:33.263976 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.263740 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b357d8d-9624-46e8-86bf-d046d06fee50-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:37:33.607835 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.607795 5615 generic.go:356] "Generic (PLEG): container finished" podID="4b357d8d-9624-46e8-86bf-d046d06fee50" containerID="50cfe6497a22759d3f52657b444efc7b0be4e5bfc9d92ad0bf6d01dd07b72126" exitCode=0 Nov 03 21:37:33.608042 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.607853 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" Nov 03 21:37:33.608042 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.607882 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" event={"ID":"4b357d8d-9624-46e8-86bf-d046d06fee50","Type":"ContainerDied","Data":"50cfe6497a22759d3f52657b444efc7b0be4e5bfc9d92ad0bf6d01dd07b72126"} Nov 03 21:37:33.608042 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.607940 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf" event={"ID":"4b357d8d-9624-46e8-86bf-d046d06fee50","Type":"ContainerDied","Data":"5126cad0a0cd78b124c2ca317c05c05813e6528943ac916ed01e47cc6624496e"} Nov 03 21:37:33.608042 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.607966 5615 scope.go:122] "RemoveContainer" containerID="50cfe6497a22759d3f52657b444efc7b0be4e5bfc9d92ad0bf6d01dd07b72126" Nov 03 21:37:33.618905 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.618886 5615 scope.go:122] "RemoveContainer" containerID="50cfe6497a22759d3f52657b444efc7b0be4e5bfc9d92ad0bf6d01dd07b72126" Nov 03 21:37:33.619234 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:37:33.619214 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50cfe6497a22759d3f52657b444efc7b0be4e5bfc9d92ad0bf6d01dd07b72126\": container with ID starting with 50cfe6497a22759d3f52657b444efc7b0be4e5bfc9d92ad0bf6d01dd07b72126 not found: ID does not exist" containerID="50cfe6497a22759d3f52657b444efc7b0be4e5bfc9d92ad0bf6d01dd07b72126" Nov 03 21:37:33.619298 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.619241 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50cfe6497a22759d3f52657b444efc7b0be4e5bfc9d92ad0bf6d01dd07b72126"} err="failed to get container status \"50cfe6497a22759d3f52657b444efc7b0be4e5bfc9d92ad0bf6d01dd07b72126\": rpc error: code = NotFound desc = could not find container \"50cfe6497a22759d3f52657b444efc7b0be4e5bfc9d92ad0bf6d01dd07b72126\": container with ID starting with 50cfe6497a22759d3f52657b444efc7b0be4e5bfc9d92ad0bf6d01dd07b72126 not found: ID does not exist" Nov 03 21:37:33.769731 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.769682 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf"] Nov 03 21:37:33.816333 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:33.816299 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-g4wjf"] Nov 03 21:37:35.652376 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:37:35.652341 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b357d8d-9624-46e8-86bf-d046d06fee50" path="/var/lib/kubelet/pods/4b357d8d-9624-46e8-86bf-d046d06fee50/volumes" Nov 03 21:37:40.620305 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:37:40.620259 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:38:10.620063 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:38:10.619971 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:38:40.619796 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:38:40.619752 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:38:46.561110 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:46.561043 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc"] Nov 03 21:38:46.561803 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:46.561552 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" podUID="fa57678a-ec54-42d8-976c-cdaed5ee04f0" containerName="istio-proxy" containerID="cri-o://676e2813dd1cc8e14f473ad99bc30595da9452074ef92da99b067647e7138330" gracePeriod=30 Nov 03 21:38:50.153434 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:50.153389 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:38:50.153888 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:50.153465 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" podUID="fa57678a-ec54-42d8-976c-cdaed5ee04f0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:38:51.844004 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:51.843937 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:38:51.954292 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:51.953978 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa57678a-ec54-42d8-976c-cdaed5ee04f0-kube-api-access-x8gvc\" (UniqueName: \"kubernetes.io/projected/fa57678a-ec54-42d8-976c-cdaed5ee04f0-kube-api-access-x8gvc\") pod \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " Nov 03 21:38:51.954292 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:51.954067 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-workload-certs\") pod \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " Nov 03 21:38:51.954292 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:51.954194 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istio-podinfo\") pod \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " Nov 03 21:38:51.954710 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:51.954678 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istio-data\") pod \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " Nov 03 21:38:51.955284 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:51.954777 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istio-token\" (UniqueName: \"kubernetes.io/projected/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istio-token\") pod \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " Nov 03 21:38:51.955284 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:51.954907 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istio-envoy\") pod \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " Nov 03 21:38:51.955284 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:51.954954 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-workload-socket\") pod \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " Nov 03 21:38:51.955284 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:51.955013 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-credential-socket\") pod \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " Nov 03 21:38:51.955284 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:51.955055 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istiod-ca-cert\") pod \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\" (UID: \"fa57678a-ec54-42d8-976c-cdaed5ee04f0\") " Nov 03 21:38:51.956439 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:51.956214 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-workload-certs" pod "fa57678a-ec54-42d8-976c-cdaed5ee04f0" (UID: "fa57678a-ec54-42d8-976c-cdaed5ee04f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:38:51.956439 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:51.956221 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istiod-ca-cert" pod "fa57678a-ec54-42d8-976c-cdaed5ee04f0" (UID: "fa57678a-ec54-42d8-976c-cdaed5ee04f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:38:51.956595 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:51.956517 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istio-data" pod "fa57678a-ec54-42d8-976c-cdaed5ee04f0" (UID: "fa57678a-ec54-42d8-976c-cdaed5ee04f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:38:51.956926 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:51.956891 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-credential-socket" pod "fa57678a-ec54-42d8-976c-cdaed5ee04f0" (UID: "fa57678a-ec54-42d8-976c-cdaed5ee04f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:38:51.957114 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:51.957063 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-workload-socket" pod "fa57678a-ec54-42d8-976c-cdaed5ee04f0" (UID: "fa57678a-ec54-42d8-976c-cdaed5ee04f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:38:51.960363 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:51.960312 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istio-envoy" pod "fa57678a-ec54-42d8-976c-cdaed5ee04f0" (UID: "fa57678a-ec54-42d8-976c-cdaed5ee04f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:38:51.960363 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:51.960329 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istio-token" pod "fa57678a-ec54-42d8-976c-cdaed5ee04f0" (UID: "fa57678a-ec54-42d8-976c-cdaed5ee04f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:38:51.960363 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:51.960350 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa57678a-ec54-42d8-976c-cdaed5ee04f0-kube-api-access-x8gvc" pod "fa57678a-ec54-42d8-976c-cdaed5ee04f0" (UID: "fa57678a-ec54-42d8-976c-cdaed5ee04f0"). InnerVolumeSpecName "kube-api-access-x8gvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:38:51.962187 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:51.962162 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istio-podinfo" pod "fa57678a-ec54-42d8-976c-cdaed5ee04f0" (UID: "fa57678a-ec54-42d8-976c-cdaed5ee04f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:38:52.057443 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:52.057403 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8gvc\" (UniqueName: \"kubernetes.io/projected/fa57678a-ec54-42d8-976c-cdaed5ee04f0-kube-api-access-x8gvc\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:52.057443 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:52.057436 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:52.057443 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:52.057445 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:52.057693 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:52.057454 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:52.057693 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:52.057463 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:52.057693 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:52.057471 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:52.057693 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:52.057480 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:52.057693 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:52.057488 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa57678a-ec54-42d8-976c-cdaed5ee04f0-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:52.057693 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:52.057498 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa57678a-ec54-42d8-976c-cdaed5ee04f0-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:52.140130 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:52.140023 5615 generic.go:356] "Generic (PLEG): container finished" podID="fa57678a-ec54-42d8-976c-cdaed5ee04f0" containerID="676e2813dd1cc8e14f473ad99bc30595da9452074ef92da99b067647e7138330" exitCode=0 Nov 03 21:38:52.140130 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:52.140091 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" Nov 03 21:38:52.140130 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:52.140116 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" event={"ID":"fa57678a-ec54-42d8-976c-cdaed5ee04f0","Type":"ContainerDied","Data":"676e2813dd1cc8e14f473ad99bc30595da9452074ef92da99b067647e7138330"} Nov 03 21:38:52.140370 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:52.140176 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc" event={"ID":"fa57678a-ec54-42d8-976c-cdaed5ee04f0","Type":"ContainerDied","Data":"ab509f395a8cf95a33d37b9a7e89744e0539ca3065b74422e5c24e87e29dbb02"} Nov 03 21:38:52.140370 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:52.140200 5615 scope.go:122] "RemoveContainer" containerID="676e2813dd1cc8e14f473ad99bc30595da9452074ef92da99b067647e7138330" Nov 03 21:38:52.151182 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:52.151159 5615 scope.go:122] "RemoveContainer" containerID="676e2813dd1cc8e14f473ad99bc30595da9452074ef92da99b067647e7138330" Nov 03 21:38:52.151515 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:38:52.151492 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"676e2813dd1cc8e14f473ad99bc30595da9452074ef92da99b067647e7138330\": container with ID starting with 676e2813dd1cc8e14f473ad99bc30595da9452074ef92da99b067647e7138330 not found: ID does not exist" containerID="676e2813dd1cc8e14f473ad99bc30595da9452074ef92da99b067647e7138330" Nov 03 21:38:52.151613 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:52.151523 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"676e2813dd1cc8e14f473ad99bc30595da9452074ef92da99b067647e7138330"} err="failed to get container status \"676e2813dd1cc8e14f473ad99bc30595da9452074ef92da99b067647e7138330\": rpc error: code = NotFound desc = could not find container \"676e2813dd1cc8e14f473ad99bc30595da9452074ef92da99b067647e7138330\": container with ID starting with 676e2813dd1cc8e14f473ad99bc30595da9452074ef92da99b067647e7138330 not found: ID does not exist" Nov 03 21:38:52.299572 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:52.299530 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc"] Nov 03 21:38:52.304345 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:52.304316 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-8xbfc"] Nov 03 21:38:53.651938 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:38:53.651902 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa57678a-ec54-42d8-976c-cdaed5ee04f0" path="/var/lib/kubelet/pods/fa57678a-ec54-42d8-976c-cdaed5ee04f0/volumes" Nov 03 21:39:10.619824 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:39:10.619780 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:39:40.619365 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:39:40.619281 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:40:05.789422 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:05.789385 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb"] Nov 03 21:40:05.789916 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:05.789603 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" podUID="af6ed462-f4bd-4bfa-9085-a6dcb7b0003c" containerName="istio-proxy" containerID="cri-o://3cb33d7d0c36b2821f42d997490d78543037d058db7e3f62a4bc37a874ae6b81" gracePeriod=30 Nov 03 21:40:10.620305 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:40:10.620261 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:40:11.126424 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.121242 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:40:11.160575 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.160484 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-workload-certs\") pod \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " Nov 03 21:40:11.160575 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.160551 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istiod-ca-cert\") pod \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " Nov 03 21:40:11.160817 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.160601 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istio-envoy\") pod \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " Nov 03 21:40:11.160817 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.160641 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-credential-socket\") pod \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " Nov 03 21:40:11.160817 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.160752 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-kube-api-access-t5sbh\" (UniqueName: \"kubernetes.io/projected/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-kube-api-access-t5sbh\") pod \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " Nov 03 21:40:11.160817 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.160757 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-workload-certs" pod "af6ed462-f4bd-4bfa-9085-a6dcb7b0003c" (UID: "af6ed462-f4bd-4bfa-9085-a6dcb7b0003c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:40:11.160817 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.160792 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-workload-socket\") pod \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " Nov 03 21:40:11.161163 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.160835 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istio-podinfo\") pod \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " Nov 03 21:40:11.161163 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.160917 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istio-token\" (UniqueName: \"kubernetes.io/projected/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istio-token\") pod \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " Nov 03 21:40:11.161163 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.160954 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istio-data\") pod \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\" (UID: \"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c\") " Nov 03 21:40:11.161163 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.160956 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-credential-socket" pod "af6ed462-f4bd-4bfa-9085-a6dcb7b0003c" (UID: "af6ed462-f4bd-4bfa-9085-a6dcb7b0003c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:40:11.161163 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.161038 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-workload-socket" pod "af6ed462-f4bd-4bfa-9085-a6dcb7b0003c" (UID: "af6ed462-f4bd-4bfa-9085-a6dcb7b0003c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:40:11.161163 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.161095 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istiod-ca-cert" pod "af6ed462-f4bd-4bfa-9085-a6dcb7b0003c" (UID: "af6ed462-f4bd-4bfa-9085-a6dcb7b0003c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:40:11.161489 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.161329 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istio-data" pod "af6ed462-f4bd-4bfa-9085-a6dcb7b0003c" (UID: "af6ed462-f4bd-4bfa-9085-a6dcb7b0003c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:40:11.161692 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.161667 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:40:11.161767 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.161697 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:40:11.161767 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.161714 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:40:11.161767 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.161730 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:40:11.161767 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.161744 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:40:11.163059 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.163033 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istio-envoy" pod "af6ed462-f4bd-4bfa-9085-a6dcb7b0003c" (UID: "af6ed462-f4bd-4bfa-9085-a6dcb7b0003c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:40:11.163215 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.163141 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istio-token" pod "af6ed462-f4bd-4bfa-9085-a6dcb7b0003c" (UID: "af6ed462-f4bd-4bfa-9085-a6dcb7b0003c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:40:11.163215 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.163187 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-kube-api-access-t5sbh" pod "af6ed462-f4bd-4bfa-9085-a6dcb7b0003c" (UID: "af6ed462-f4bd-4bfa-9085-a6dcb7b0003c"). InnerVolumeSpecName "kube-api-access-t5sbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:40:11.163303 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.163248 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istio-podinfo" pod "af6ed462-f4bd-4bfa-9085-a6dcb7b0003c" (UID: "af6ed462-f4bd-4bfa-9085-a6dcb7b0003c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:40:11.262815 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.262766 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:40:11.262815 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.262805 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:40:11.262815 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.262817 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5sbh\" (UniqueName: \"kubernetes.io/projected/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-kube-api-access-t5sbh\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:40:11.262815 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.262828 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:40:11.706023 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.705953 5615 generic.go:356] "Generic (PLEG): container finished" podID="af6ed462-f4bd-4bfa-9085-a6dcb7b0003c" containerID="3cb33d7d0c36b2821f42d997490d78543037d058db7e3f62a4bc37a874ae6b81" exitCode=0 Nov 03 21:40:11.706023 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.706016 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" Nov 03 21:40:11.706599 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.706022 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" event={"ID":"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c","Type":"ContainerDied","Data":"3cb33d7d0c36b2821f42d997490d78543037d058db7e3f62a4bc37a874ae6b81"} Nov 03 21:40:11.706599 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.706103 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb" event={"ID":"af6ed462-f4bd-4bfa-9085-a6dcb7b0003c","Type":"ContainerDied","Data":"c5a0c4143b9762b1569e43500a4727506974437e734ad6a2e6fe65854db59a3b"} Nov 03 21:40:11.706599 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.706134 5615 scope.go:122] "RemoveContainer" containerID="3cb33d7d0c36b2821f42d997490d78543037d058db7e3f62a4bc37a874ae6b81" Nov 03 21:40:11.717209 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.717185 5615 scope.go:122] "RemoveContainer" containerID="8b43aa028e5711c209c003852abc417446b84ec7da1205fd1260a477c808e5c3" Nov 03 21:40:11.727380 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.727354 5615 scope.go:122] "RemoveContainer" containerID="3cb33d7d0c36b2821f42d997490d78543037d058db7e3f62a4bc37a874ae6b81" Nov 03 21:40:11.727711 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:40:11.727689 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cb33d7d0c36b2821f42d997490d78543037d058db7e3f62a4bc37a874ae6b81\": container with ID starting with 3cb33d7d0c36b2821f42d997490d78543037d058db7e3f62a4bc37a874ae6b81 not found: ID does not exist" containerID="3cb33d7d0c36b2821f42d997490d78543037d058db7e3f62a4bc37a874ae6b81" Nov 03 21:40:11.727769 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.727724 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cb33d7d0c36b2821f42d997490d78543037d058db7e3f62a4bc37a874ae6b81"} err="failed to get container status \"3cb33d7d0c36b2821f42d997490d78543037d058db7e3f62a4bc37a874ae6b81\": rpc error: code = NotFound desc = could not find container \"3cb33d7d0c36b2821f42d997490d78543037d058db7e3f62a4bc37a874ae6b81\": container with ID starting with 3cb33d7d0c36b2821f42d997490d78543037d058db7e3f62a4bc37a874ae6b81 not found: ID does not exist" Nov 03 21:40:11.727769 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.727745 5615 scope.go:122] "RemoveContainer" containerID="8b43aa028e5711c209c003852abc417446b84ec7da1205fd1260a477c808e5c3" Nov 03 21:40:11.728029 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:40:11.728012 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b43aa028e5711c209c003852abc417446b84ec7da1205fd1260a477c808e5c3\": container with ID starting with 8b43aa028e5711c209c003852abc417446b84ec7da1205fd1260a477c808e5c3 not found: ID does not exist" containerID="8b43aa028e5711c209c003852abc417446b84ec7da1205fd1260a477c808e5c3" Nov 03 21:40:11.728139 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.728033 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b43aa028e5711c209c003852abc417446b84ec7da1205fd1260a477c808e5c3"} err="failed to get container status \"8b43aa028e5711c209c003852abc417446b84ec7da1205fd1260a477c808e5c3\": rpc error: code = NotFound desc = could not find container \"8b43aa028e5711c209c003852abc417446b84ec7da1205fd1260a477c808e5c3\": container with ID starting with 8b43aa028e5711c209c003852abc417446b84ec7da1205fd1260a477c808e5c3 not found: ID does not exist" Nov 03 21:40:11.863110 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.863061 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb"] Nov 03 21:40:11.910267 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:11.910186 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-b22936-openshift-default-8475879b7c-mgxpb"] Nov 03 21:40:13.652087 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:13.652039 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af6ed462-f4bd-4bfa-9085-a6dcb7b0003c" path="/var/lib/kubelet/pods/af6ed462-f4bd-4bfa-9085-a6dcb7b0003c/volumes" Nov 03 21:40:26.198661 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:26.198627 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j"] Nov 03 21:40:40.619531 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:40:40.619490 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:40:41.219427 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:41.219380 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" podUID="fd483f7c-f5d6-427f-a946-844ac94c9d9e" containerName="main" containerID="cri-o://af48698100bc7efb6235af2692fb20dd7bd8c41cd90614c8014bb8c84ff8ce00" gracePeriod=45 Nov 03 21:40:41.489706 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:41.489673 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" Nov 03 21:40:41.572446 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:41.572409 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd483f7c-f5d6-427f-a946-844ac94c9d9e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd483f7c-f5d6-427f-a946-844ac94c9d9e-model-cache\") pod \"fd483f7c-f5d6-427f-a946-844ac94c9d9e\" (UID: \"fd483f7c-f5d6-427f-a946-844ac94c9d9e\") " Nov 03 21:40:41.572644 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:41.572475 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd483f7c-f5d6-427f-a946-844ac94c9d9e-kube-api-access-kljlg\" (UniqueName: \"kubernetes.io/projected/fd483f7c-f5d6-427f-a946-844ac94c9d9e-kube-api-access-kljlg\") pod \"fd483f7c-f5d6-427f-a946-844ac94c9d9e\" (UID: \"fd483f7c-f5d6-427f-a946-844ac94c9d9e\") " Nov 03 21:40:41.572644 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:41.572534 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd483f7c-f5d6-427f-a946-844ac94c9d9e-home\" (UniqueName: \"kubernetes.io/empty-dir/fd483f7c-f5d6-427f-a946-844ac94c9d9e-home\") pod \"fd483f7c-f5d6-427f-a946-844ac94c9d9e\" (UID: \"fd483f7c-f5d6-427f-a946-844ac94c9d9e\") " Nov 03 21:40:41.572644 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:41.572591 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd483f7c-f5d6-427f-a946-844ac94c9d9e-tls-certs\" (UniqueName: \"kubernetes.io/secret/fd483f7c-f5d6-427f-a946-844ac94c9d9e-tls-certs\") pod \"fd483f7c-f5d6-427f-a946-844ac94c9d9e\" (UID: \"fd483f7c-f5d6-427f-a946-844ac94c9d9e\") " Nov 03 21:40:41.572644 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:41.572639 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd483f7c-f5d6-427f-a946-844ac94c9d9e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd483f7c-f5d6-427f-a946-844ac94c9d9e-tmp-dir\") pod \"fd483f7c-f5d6-427f-a946-844ac94c9d9e\" (UID: \"fd483f7c-f5d6-427f-a946-844ac94c9d9e\") " Nov 03 21:40:41.572879 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:41.572701 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd483f7c-f5d6-427f-a946-844ac94c9d9e-model-cache" pod "fd483f7c-f5d6-427f-a946-844ac94c9d9e" (UID: "fd483f7c-f5d6-427f-a946-844ac94c9d9e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:40:41.572879 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:41.572784 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd483f7c-f5d6-427f-a946-844ac94c9d9e-home" pod "fd483f7c-f5d6-427f-a946-844ac94c9d9e" (UID: "fd483f7c-f5d6-427f-a946-844ac94c9d9e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:40:41.572879 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:41.572808 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd483f7c-f5d6-427f-a946-844ac94c9d9e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd483f7c-f5d6-427f-a946-844ac94c9d9e-dshm\") pod \"fd483f7c-f5d6-427f-a946-844ac94c9d9e\" (UID: \"fd483f7c-f5d6-427f-a946-844ac94c9d9e\") " Nov 03 21:40:41.573030 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:41.572926 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd483f7c-f5d6-427f-a946-844ac94c9d9e-tmp-dir" pod "fd483f7c-f5d6-427f-a946-844ac94c9d9e" (UID: "fd483f7c-f5d6-427f-a946-844ac94c9d9e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:40:41.573294 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:41.573275 5615 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd483f7c-f5d6-427f-a946-844ac94c9d9e-model-cache\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:40:41.573399 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:41.573297 5615 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd483f7c-f5d6-427f-a946-844ac94c9d9e-home\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:40:41.573399 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:41.573312 5615 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd483f7c-f5d6-427f-a946-844ac94c9d9e-tmp-dir\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:40:41.574827 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:41.574807 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd483f7c-f5d6-427f-a946-844ac94c9d9e-dshm" pod "fd483f7c-f5d6-427f-a946-844ac94c9d9e" (UID: "fd483f7c-f5d6-427f-a946-844ac94c9d9e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:40:41.574903 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:41.574835 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd483f7c-f5d6-427f-a946-844ac94c9d9e-tls-certs" pod "fd483f7c-f5d6-427f-a946-844ac94c9d9e" (UID: "fd483f7c-f5d6-427f-a946-844ac94c9d9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:40:41.574945 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:41.574935 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd483f7c-f5d6-427f-a946-844ac94c9d9e-kube-api-access-kljlg" pod "fd483f7c-f5d6-427f-a946-844ac94c9d9e" (UID: "fd483f7c-f5d6-427f-a946-844ac94c9d9e"). InnerVolumeSpecName "kube-api-access-kljlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:40:41.673975 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:41.673939 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kljlg\" (UniqueName: \"kubernetes.io/projected/fd483f7c-f5d6-427f-a946-844ac94c9d9e-kube-api-access-kljlg\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:40:41.673975 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:41.673968 5615 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd483f7c-f5d6-427f-a946-844ac94c9d9e-tls-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:40:41.673975 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:41.673978 5615 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd483f7c-f5d6-427f-a946-844ac94c9d9e-dshm\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:40:41.888615 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:41.888527 5615 generic.go:356] "Generic (PLEG): container finished" podID="fd483f7c-f5d6-427f-a946-844ac94c9d9e" containerID="af48698100bc7efb6235af2692fb20dd7bd8c41cd90614c8014bb8c84ff8ce00" exitCode=0 Nov 03 21:40:41.888615 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:41.888580 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" Nov 03 21:40:41.888816 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:41.888614 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" event={"ID":"fd483f7c-f5d6-427f-a946-844ac94c9d9e","Type":"ContainerDied","Data":"af48698100bc7efb6235af2692fb20dd7bd8c41cd90614c8014bb8c84ff8ce00"} Nov 03 21:40:41.888816 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:41.888657 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" event={"ID":"fd483f7c-f5d6-427f-a946-844ac94c9d9e","Type":"ContainerDied","Data":"075316b55f4a75222bf7022ef51fb92826a7325d7aee49c7c122f7c2b5ff9e62"} Nov 03 21:40:41.888816 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:41.888674 5615 scope.go:122] "RemoveContainer" containerID="af48698100bc7efb6235af2692fb20dd7bd8c41cd90614c8014bb8c84ff8ce00" Nov 03 21:40:41.901563 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:41.901542 5615 scope.go:122] "RemoveContainer" containerID="af48698100bc7efb6235af2692fb20dd7bd8c41cd90614c8014bb8c84ff8ce00" Nov 03 21:40:41.901968 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:40:41.901946 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af48698100bc7efb6235af2692fb20dd7bd8c41cd90614c8014bb8c84ff8ce00\": container with ID starting with af48698100bc7efb6235af2692fb20dd7bd8c41cd90614c8014bb8c84ff8ce00 not found: ID does not exist" containerID="af48698100bc7efb6235af2692fb20dd7bd8c41cd90614c8014bb8c84ff8ce00" Nov 03 21:40:41.902047 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:41.901977 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af48698100bc7efb6235af2692fb20dd7bd8c41cd90614c8014bb8c84ff8ce00"} err="failed to get container status \"af48698100bc7efb6235af2692fb20dd7bd8c41cd90614c8014bb8c84ff8ce00\": rpc error: code = NotFound desc = could not find container \"af48698100bc7efb6235af2692fb20dd7bd8c41cd90614c8014bb8c84ff8ce00\": container with ID starting with af48698100bc7efb6235af2692fb20dd7bd8c41cd90614c8014bb8c84ff8ce00 not found: ID does not exist" Nov 03 21:40:42.050834 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:42.050787 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j"] Nov 03 21:40:42.054947 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:42.054911 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j"] Nov 03 21:40:42.437822 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:42.437778 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-9818d8/auth-test-model-9818d8-kserve-5968dbccd9-cgn8j" podUID="fd483f7c-f5d6-427f-a946-844ac94c9d9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.167:8000/ready\": context deadline exceeded" Nov 03 21:40:43.652176 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:40:43.652140 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd483f7c-f5d6-427f-a946-844ac94c9d9e" path="/var/lib/kubelet/pods/fd483f7c-f5d6-427f-a946-844ac94c9d9e/volumes" Nov 03 21:41:10.619031 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:41:10.618943 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:41:13.564388 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.564352 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w"] Nov 03 21:41:13.565085 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.565057 5615 cpu_manager.go:414] "containerMap: removing container" podUID="af6ed462-f4bd-4bfa-9085-a6dcb7b0003c" containerName="istio-proxy" Nov 03 21:41:13.565164 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.565113 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af6ed462-f4bd-4bfa-9085-a6dcb7b0003c" containerName="istio-proxy" Nov 03 21:41:13.565164 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.565132 5615 cpu_manager.go:414] "containerMap: removing container" podUID="af6ed462-f4bd-4bfa-9085-a6dcb7b0003c" containerName="istio-proxy" Nov 03 21:41:13.565164 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.565137 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af6ed462-f4bd-4bfa-9085-a6dcb7b0003c" containerName="istio-proxy" Nov 03 21:41:13.565164 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.565153 5615 cpu_manager.go:414] "containerMap: removing container" podUID="fa57678a-ec54-42d8-976c-cdaed5ee04f0" containerName="istio-proxy" Nov 03 21:41:13.565164 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.565158 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa57678a-ec54-42d8-976c-cdaed5ee04f0" containerName="istio-proxy" Nov 03 21:41:13.565330 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.565169 5615 cpu_manager.go:414] "containerMap: removing container" podUID="4b357d8d-9624-46e8-86bf-d046d06fee50" containerName="istio-proxy" Nov 03 21:41:13.565330 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.565174 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b357d8d-9624-46e8-86bf-d046d06fee50" containerName="istio-proxy" Nov 03 21:41:13.565330 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.565191 5615 cpu_manager.go:414] "containerMap: removing container" podUID="fd483f7c-f5d6-427f-a946-844ac94c9d9e" containerName="main" Nov 03 21:41:13.565330 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.565199 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd483f7c-f5d6-427f-a946-844ac94c9d9e" containerName="main" Nov 03 21:41:13.565330 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.565210 5615 cpu_manager.go:414] "containerMap: removing container" podUID="7242ebfd-977b-4b38-9c9f-f9c125ef3280" containerName="istio-proxy" Nov 03 21:41:13.565330 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.565217 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7242ebfd-977b-4b38-9c9f-f9c125ef3280" containerName="istio-proxy" Nov 03 21:41:13.565330 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.565228 5615 cpu_manager.go:414] "containerMap: removing container" podUID="1357b2a2-d12d-458e-942a-bdfc808972c1" containerName="istio-proxy" Nov 03 21:41:13.565330 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.565233 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1357b2a2-d12d-458e-942a-bdfc808972c1" containerName="istio-proxy" Nov 03 21:41:13.565330 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.565327 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd483f7c-f5d6-427f-a946-844ac94c9d9e" containerName="main" Nov 03 21:41:13.565665 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.565336 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="7242ebfd-977b-4b38-9c9f-f9c125ef3280" containerName="istio-proxy" Nov 03 21:41:13.565665 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.565346 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="1357b2a2-d12d-458e-942a-bdfc808972c1" containerName="istio-proxy" Nov 03 21:41:13.565665 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.565360 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="af6ed462-f4bd-4bfa-9085-a6dcb7b0003c" containerName="istio-proxy" Nov 03 21:41:13.565665 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.565366 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="af6ed462-f4bd-4bfa-9085-a6dcb7b0003c" containerName="istio-proxy" Nov 03 21:41:13.565665 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.565374 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa57678a-ec54-42d8-976c-cdaed5ee04f0" containerName="istio-proxy" Nov 03 21:41:13.565665 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.565380 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b357d8d-9624-46e8-86bf-d046d06fee50" containerName="istio-proxy" Nov 03 21:41:13.568927 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.568908 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w" Nov 03 21:41:13.609351 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.609307 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26zsn\" (UniqueName: \"kubernetes.io/projected/d8f75aac-8457-4abc-ad9f-6cbb8f462f6b-kube-api-access-26zsn\") pod \"payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w\" (UID: \"d8f75aac-8457-4abc-ad9f-6cbb8f462f6b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w" Nov 03 21:41:13.609545 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.609394 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8f75aac-8457-4abc-ad9f-6cbb8f462f6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w\" (UID: \"d8f75aac-8457-4abc-ad9f-6cbb8f462f6b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w" Nov 03 21:41:13.687059 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.686167 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-a3dc77\"" Nov 03 21:41:13.687059 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.686280 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-a3dc77-dockercfg-krd79\"" Nov 03 21:41:13.709460 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.709421 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w"] Nov 03 21:41:13.714227 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.712912 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26zsn\" (UniqueName: \"kubernetes.io/projected/d8f75aac-8457-4abc-ad9f-6cbb8f462f6b-kube-api-access-26zsn\") pod \"payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w\" (UID: \"d8f75aac-8457-4abc-ad9f-6cbb8f462f6b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w" Nov 03 21:41:13.714521 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.714501 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8f75aac-8457-4abc-ad9f-6cbb8f462f6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w\" (UID: \"d8f75aac-8457-4abc-ad9f-6cbb8f462f6b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w" Nov 03 21:41:13.722185 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.722153 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8f75aac-8457-4abc-ad9f-6cbb8f462f6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w\" (UID: \"d8f75aac-8457-4abc-ad9f-6cbb8f462f6b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w" Nov 03 21:41:13.771433 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.771402 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26zsn\" (UniqueName: \"kubernetes.io/projected/d8f75aac-8457-4abc-ad9f-6cbb8f462f6b-kube-api-access-26zsn\") pod \"payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w\" (UID: \"d8f75aac-8457-4abc-ad9f-6cbb8f462f6b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w" Nov 03 21:41:13.882600 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:13.882508 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w" Nov 03 21:41:14.254617 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:14.254577 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w"] Nov 03 21:41:14.255713 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:41:14.255681 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f75aac_8457_4abc_ad9f_6cbb8f462f6b.slice/crio-577e7f9b09dd14df816034ff5d48e538fd19007151ef20a376324ae3e02a28a1 WatchSource:0}: Error finding container 577e7f9b09dd14df816034ff5d48e538fd19007151ef20a376324ae3e02a28a1: Status 404 returned error can't find the container with id 577e7f9b09dd14df816034ff5d48e538fd19007151ef20a376324ae3e02a28a1 Nov 03 21:41:14.464939 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:14.464900 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx"] Nov 03 21:41:14.465156 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:14.465119 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" podUID="d837c860-39f5-4cf0-b6f3-ab441ee9e580" containerName="istio-proxy" containerID="cri-o://01142474b9e27ab0d5602c198f34c83574c5124f788b7696a9cd1535cc904727" gracePeriod=30 Nov 03 21:41:15.137871 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:15.137821 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w" event={"ID":"d8f75aac-8457-4abc-ad9f-6cbb8f462f6b","Type":"ContainerStarted","Data":"13eec92495be8ebec49db0dcb7cefd6b775c9dbdb5a0119ef6e41571ab9eb627"} Nov 03 21:41:15.138322 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:15.137896 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w" Nov 03 21:41:15.138322 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:15.137921 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w" event={"ID":"d8f75aac-8457-4abc-ad9f-6cbb8f462f6b","Type":"ContainerStarted","Data":"577e7f9b09dd14df816034ff5d48e538fd19007151ef20a376324ae3e02a28a1"} Nov 03 21:41:15.250756 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:15.250700 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w" podStartSLOduration=2.250685052 podStartE2EDuration="2.250685052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:41:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:41:15.247963583 +0000 UTC m=+3169.715742307" watchObservedRunningTime="2026-11-03 21:41:15.250685052 +0000 UTC m=+3169.718463783" Nov 03 21:41:16.208842 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:16.208805 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:41:16.209573 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:16.209538 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" podUID="d837c860-39f5-4cf0-b6f3-ab441ee9e580" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:41:19.732329 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:19.732257 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:41:19.803637 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:19.803603 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-workload-certs\") pod \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " Nov 03 21:41:19.803831 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:19.803662 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istio-podinfo\") pod \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " Nov 03 21:41:19.803831 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:19.803734 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d837c860-39f5-4cf0-b6f3-ab441ee9e580-kube-api-access-dswlw\" (UniqueName: \"kubernetes.io/projected/d837c860-39f5-4cf0-b6f3-ab441ee9e580-kube-api-access-dswlw\") pod \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " Nov 03 21:41:19.803952 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:19.803829 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istio-data\") pod \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " Nov 03 21:41:19.803952 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:19.803863 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istio-envoy\") pod \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " Nov 03 21:41:19.803952 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:19.803897 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-credential-socket\") pod \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " Nov 03 21:41:19.803952 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:19.803925 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istiod-ca-cert\") pod \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " Nov 03 21:41:19.803952 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:19.803926 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-workload-certs" pod "d837c860-39f5-4cf0-b6f3-ab441ee9e580" (UID: "d837c860-39f5-4cf0-b6f3-ab441ee9e580"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:19.804339 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:19.803998 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-workload-socket\") pod \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " Nov 03 21:41:19.804339 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:19.804044 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istio-token\" (UniqueName: \"kubernetes.io/projected/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istio-token\") pod \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\" (UID: \"d837c860-39f5-4cf0-b6f3-ab441ee9e580\") " Nov 03 21:41:19.804339 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:19.804207 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-credential-socket" pod "d837c860-39f5-4cf0-b6f3-ab441ee9e580" (UID: "d837c860-39f5-4cf0-b6f3-ab441ee9e580"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:19.804339 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:19.804303 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-workload-socket" pod "d837c860-39f5-4cf0-b6f3-ab441ee9e580" (UID: "d837c860-39f5-4cf0-b6f3-ab441ee9e580"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:19.804556 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:19.804384 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istiod-ca-cert" pod "d837c860-39f5-4cf0-b6f3-ab441ee9e580" (UID: "d837c860-39f5-4cf0-b6f3-ab441ee9e580"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:41:19.804556 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:19.804436 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istio-data" pod "d837c860-39f5-4cf0-b6f3-ab441ee9e580" (UID: "d837c860-39f5-4cf0-b6f3-ab441ee9e580"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:19.804663 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:19.804637 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:41:19.804663 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:19.804651 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:41:19.804663 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:19.804660 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:41:19.804827 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:19.804669 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:41:19.804827 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:19.804677 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:41:19.806217 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:19.806186 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istio-podinfo" pod "d837c860-39f5-4cf0-b6f3-ab441ee9e580" (UID: "d837c860-39f5-4cf0-b6f3-ab441ee9e580"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:41:19.806351 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:19.806257 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istio-envoy" pod "d837c860-39f5-4cf0-b6f3-ab441ee9e580" (UID: "d837c860-39f5-4cf0-b6f3-ab441ee9e580"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:19.806497 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:19.806474 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d837c860-39f5-4cf0-b6f3-ab441ee9e580-kube-api-access-dswlw" pod "d837c860-39f5-4cf0-b6f3-ab441ee9e580" (UID: "d837c860-39f5-4cf0-b6f3-ab441ee9e580"). InnerVolumeSpecName "kube-api-access-dswlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:41:19.806615 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:19.806597 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istio-token" pod "d837c860-39f5-4cf0-b6f3-ab441ee9e580" (UID: "d837c860-39f5-4cf0-b6f3-ab441ee9e580"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:41:19.905663 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:19.905625 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:41:19.905663 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:19.905657 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:41:19.905663 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:19.905666 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d837c860-39f5-4cf0-b6f3-ab441ee9e580-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:41:19.905913 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:19.905676 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dswlw\" (UniqueName: \"kubernetes.io/projected/d837c860-39f5-4cf0-b6f3-ab441ee9e580-kube-api-access-dswlw\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:41:20.170771 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:20.170733 5615 generic.go:356] "Generic (PLEG): container finished" podID="d837c860-39f5-4cf0-b6f3-ab441ee9e580" containerID="01142474b9e27ab0d5602c198f34c83574c5124f788b7696a9cd1535cc904727" exitCode=0 Nov 03 21:41:20.170985 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:20.170784 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" Nov 03 21:41:20.170985 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:20.170817 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" event={"ID":"d837c860-39f5-4cf0-b6f3-ab441ee9e580","Type":"ContainerDied","Data":"01142474b9e27ab0d5602c198f34c83574c5124f788b7696a9cd1535cc904727"} Nov 03 21:41:20.170985 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:20.170877 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx" event={"ID":"d837c860-39f5-4cf0-b6f3-ab441ee9e580","Type":"ContainerDied","Data":"fbe7e66dc48361459171b6f50ce9617ec0e602d2650be3eee499d6210b3b44d9"} Nov 03 21:41:20.170985 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:20.170907 5615 scope.go:122] "RemoveContainer" containerID="01142474b9e27ab0d5602c198f34c83574c5124f788b7696a9cd1535cc904727" Nov 03 21:41:20.181996 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:20.181976 5615 scope.go:122] "RemoveContainer" containerID="01142474b9e27ab0d5602c198f34c83574c5124f788b7696a9cd1535cc904727" Nov 03 21:41:20.182351 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:41:20.182318 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01142474b9e27ab0d5602c198f34c83574c5124f788b7696a9cd1535cc904727\": container with ID starting with 01142474b9e27ab0d5602c198f34c83574c5124f788b7696a9cd1535cc904727 not found: ID does not exist" containerID="01142474b9e27ab0d5602c198f34c83574c5124f788b7696a9cd1535cc904727" Nov 03 21:41:20.182482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:20.182349 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01142474b9e27ab0d5602c198f34c83574c5124f788b7696a9cd1535cc904727"} err="failed to get container status \"01142474b9e27ab0d5602c198f34c83574c5124f788b7696a9cd1535cc904727\": rpc error: code = NotFound desc = could not find container \"01142474b9e27ab0d5602c198f34c83574c5124f788b7696a9cd1535cc904727\": container with ID starting with 01142474b9e27ab0d5602c198f34c83574c5124f788b7696a9cd1535cc904727 not found: ID does not exist" Nov 03 21:41:20.332424 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:20.332391 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx"] Nov 03 21:41:20.379659 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:20.379626 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-c2rhx"] Nov 03 21:41:21.652356 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:21.652320 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d837c860-39f5-4cf0-b6f3-ab441ee9e580" path="/var/lib/kubelet/pods/d837c860-39f5-4cf0-b6f3-ab441ee9e580/volumes" Nov 03 21:41:26.147555 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:41:26.147523 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w" Nov 03 21:41:40.619291 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:41:40.619248 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:42:10.620259 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:42:10.620215 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:42:12.836556 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:12.836514 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9"] Nov 03 21:42:12.837137 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:12.837122 5615 cpu_manager.go:414] "containerMap: removing container" podUID="d837c860-39f5-4cf0-b6f3-ab441ee9e580" containerName="istio-proxy" Nov 03 21:42:12.837192 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:12.837138 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d837c860-39f5-4cf0-b6f3-ab441ee9e580" containerName="istio-proxy" Nov 03 21:42:12.837239 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:12.837230 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="d837c860-39f5-4cf0-b6f3-ab441ee9e580" containerName="istio-proxy" Nov 03 21:42:12.840783 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:12.840762 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9" Nov 03 21:42:12.893069 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:12.893034 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-a3dc77-serving-cert\"" Nov 03 21:42:12.942275 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:12.942239 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9"] Nov 03 21:42:12.968800 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:12.968762 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79d06746-463e-4043-872b-6f64949dccc8-metrics-certs\") pod \"maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9\" (UID: \"79d06746-463e-4043-872b-6f64949dccc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9" Nov 03 21:42:12.969009 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:12.968950 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w7cw\" (UniqueName: \"kubernetes.io/projected/79d06746-463e-4043-872b-6f64949dccc8-kube-api-access-2w7cw\") pod \"maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9\" (UID: \"79d06746-463e-4043-872b-6f64949dccc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9" Nov 03 21:42:12.969152 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:12.969130 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79d06746-463e-4043-872b-6f64949dccc8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9\" (UID: \"79d06746-463e-4043-872b-6f64949dccc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9" Nov 03 21:42:12.969288 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:12.969268 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79d06746-463e-4043-872b-6f64949dccc8-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9\" (UID: \"79d06746-463e-4043-872b-6f64949dccc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9" Nov 03 21:42:13.070464 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:13.070427 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79d06746-463e-4043-872b-6f64949dccc8-metrics-certs\") pod \"maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9\" (UID: \"79d06746-463e-4043-872b-6f64949dccc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9" Nov 03 21:42:13.070657 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:13.070546 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w7cw\" (UniqueName: \"kubernetes.io/projected/79d06746-463e-4043-872b-6f64949dccc8-kube-api-access-2w7cw\") pod \"maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9\" (UID: \"79d06746-463e-4043-872b-6f64949dccc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9" Nov 03 21:42:13.070657 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:13.070626 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79d06746-463e-4043-872b-6f64949dccc8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9\" (UID: \"79d06746-463e-4043-872b-6f64949dccc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9" Nov 03 21:42:13.070755 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:13.070715 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79d06746-463e-4043-872b-6f64949dccc8-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9\" (UID: \"79d06746-463e-4043-872b-6f64949dccc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9" Nov 03 21:42:13.070815 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:42:13.070792 5615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-serving-cert: secret "maas-api-e2e-extmod-a-a3dc77-serving-cert" not found Nov 03 21:42:13.070895 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:42:13.070872 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79d06746-463e-4043-872b-6f64949dccc8-maas-api-tls podName:79d06746-463e-4043-872b-6f64949dccc8 nodeName:}" failed. No retries permitted until 2026-11-03 21:42:13.570850893 +0000 UTC m=+3228.038629608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/79d06746-463e-4043-872b-6f64949dccc8-maas-api-tls") pod "maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9" (UID: "79d06746-463e-4043-872b-6f64949dccc8") : secret "maas-api-e2e-extmod-a-a3dc77-serving-cert" not found Nov 03 21:42:13.071352 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:13.071332 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79d06746-463e-4043-872b-6f64949dccc8-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9\" (UID: \"79d06746-463e-4043-872b-6f64949dccc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9" Nov 03 21:42:13.073123 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:13.073091 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79d06746-463e-4043-872b-6f64949dccc8-metrics-certs\") pod \"maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9\" (UID: \"79d06746-463e-4043-872b-6f64949dccc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9" Nov 03 21:42:13.133002 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:13.132923 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w7cw\" (UniqueName: \"kubernetes.io/projected/79d06746-463e-4043-872b-6f64949dccc8-kube-api-access-2w7cw\") pod \"maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9\" (UID: \"79d06746-463e-4043-872b-6f64949dccc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9" Nov 03 21:42:13.580593 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:13.580553 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79d06746-463e-4043-872b-6f64949dccc8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9\" (UID: \"79d06746-463e-4043-872b-6f64949dccc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9" Nov 03 21:42:13.583236 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:13.583192 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79d06746-463e-4043-872b-6f64949dccc8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9\" (UID: \"79d06746-463e-4043-872b-6f64949dccc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9" Nov 03 21:42:13.754907 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:13.754871 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9" Nov 03 21:42:14.130191 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:14.130150 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9"] Nov 03 21:42:14.132840 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:42:14.132807 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d06746_463e_4043_872b_6f64949dccc8.slice/crio-34c56e5d48d0637287f3167c60889993cbd3486bba00cf3b57fe620ef51e4315 WatchSource:0}: Error finding container 34c56e5d48d0637287f3167c60889993cbd3486bba00cf3b57fe620ef51e4315: Status 404 returned error can't find the container with id 34c56e5d48d0637287f3167c60889993cbd3486bba00cf3b57fe620ef51e4315 Nov 03 21:42:14.134215 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:14.134198 5615 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:42:14.510877 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:14.510838 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9" event={"ID":"79d06746-463e-4043-872b-6f64949dccc8","Type":"ContainerStarted","Data":"34c56e5d48d0637287f3167c60889993cbd3486bba00cf3b57fe620ef51e4315"} Nov 03 21:42:15.518420 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:15.518378 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9" event={"ID":"79d06746-463e-4043-872b-6f64949dccc8","Type":"ContainerStarted","Data":"66083bdfca353ea00e803cfd542ab34c1bfcd6d79328380e8e59c93aa344d898"} Nov 03 21:42:15.518824 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:15.518443 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9" Nov 03 21:42:15.633046 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:15.632981 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9" podStartSLOduration=2.699472784 podStartE2EDuration="3.632966659s" totalImagesPullingTime="933.493875ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:42:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:15.630795588 +0000 UTC m=+3230.098574323" watchObservedRunningTime="2026-11-03 21:42:15.632966659 +0000 UTC m=+3230.100745382" Nov 03 21:42:21.530412 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:21.530379 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9" Nov 03 21:42:29.493132 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.493067 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8"] Nov 03 21:42:29.501505 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.501470 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:29.554186 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.554149 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-d5e2db-openshift-default-dockercfg-ptbkq\"" Nov 03 21:42:29.629328 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.629280 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-workload-certs\") pod \"e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:29.629534 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.629367 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istio-data\") pod \"e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:29.629534 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.629425 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istio-podinfo\") pod \"e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:29.629534 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.629469 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-workload-socket\") pod \"e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:29.629695 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.629544 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istiod-ca-cert\") pod \"e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:29.629695 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.629574 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ssg8\" (UniqueName: \"kubernetes.io/projected/5b1a343a-cae6-4551-b189-bb6b0e978ba4-kube-api-access-7ssg8\") pod \"e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:29.629695 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.629607 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-credential-socket\") pod \"e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:29.629858 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.629700 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istio-envoy\") pod \"e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:29.629858 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.629728 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istio-token\") pod \"e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:29.633180 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.633144 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8"] Nov 03 21:42:29.730960 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.730913 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istio-podinfo\") pod \"e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:29.731185 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.730987 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-workload-socket\") pod \"e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:29.731185 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.731057 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istiod-ca-cert\") pod \"e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:29.731185 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.731123 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ssg8\" (UniqueName: \"kubernetes.io/projected/5b1a343a-cae6-4551-b189-bb6b0e978ba4-kube-api-access-7ssg8\") pod \"e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:29.731185 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.731161 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-credential-socket\") pod \"e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:29.731423 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.731296 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istio-envoy\") pod \"e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:29.731423 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.731337 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istio-token\") pod \"e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:29.731423 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.731394 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-workload-certs\") pod \"e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:29.731610 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.731458 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istio-data\") pod \"e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:29.731731 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.731706 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-credential-socket\") pod \"e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:29.731861 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.731834 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istio-data\") pod \"e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:29.732035 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.731899 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istiod-ca-cert\") pod \"e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:29.732167 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.731977 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-workload-socket\") pod \"e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:29.732167 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.732014 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-workload-certs\") pod \"e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:29.733838 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.733813 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istio-envoy\") pod \"e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:29.733990 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.733963 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istio-podinfo\") pod \"e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:29.789815 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.789741 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istio-token\") pod \"e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:29.789951 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.789881 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ssg8\" (UniqueName: \"kubernetes.io/projected/5b1a343a-cae6-4551-b189-bb6b0e978ba4-kube-api-access-7ssg8\") pod \"e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:29.818150 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:29.818114 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:30.257368 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:30.257338 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8"] Nov 03 21:42:30.259138 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:42:30.259107 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1a343a_cae6_4551_b189_bb6b0e978ba4.slice/crio-8fd6bc341b0a84d7011fc651fac3e54544d269f9e728de70c85681c3c7879d66 WatchSource:0}: Error finding container 8fd6bc341b0a84d7011fc651fac3e54544d269f9e728de70c85681c3c7879d66: Status 404 returned error can't find the container with id 8fd6bc341b0a84d7011fc651fac3e54544d269f9e728de70c85681c3c7879d66 Nov 03 21:42:30.262241 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:30.262202 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:42:30.262325 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:30.262274 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:42:30.262325 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:30.262308 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:42:30.695714 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:30.695621 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" event={"ID":"5b1a343a-cae6-4551-b189-bb6b0e978ba4","Type":"ContainerStarted","Data":"fc0200cb8c2785af885a6f131d47cfe4d0892b1f1850a60b68dcfcaf6e05cad5"} Nov 03 21:42:30.695714 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:30.695690 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" event={"ID":"5b1a343a-cae6-4551-b189-bb6b0e978ba4","Type":"ContainerStarted","Data":"8fd6bc341b0a84d7011fc651fac3e54544d269f9e728de70c85681c3c7879d66"} Nov 03 21:42:30.821698 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:30.819000 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:31.825802 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:31.825767 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:31.943648 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:31.943576 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" podStartSLOduration=2.943561287 podStartE2EDuration="2.943561287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:30.828181749 +0000 UTC m=+3245.295960477" watchObservedRunningTime="2026-11-03 21:42:31.943561287 +0000 UTC m=+3246.411340099" Nov 03 21:42:32.707857 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:32.707825 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:32.708763 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:32.708745 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:42:34.655280 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.655237 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp"] Nov 03 21:42:34.659754 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.659722 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:34.784640 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.784550 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-c0fc5f-openshift-default-dockercfg-blgfm\"" Nov 03 21:42:34.789026 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.788989 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp"] Nov 03 21:42:34.822331 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.820563 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istio-data\") pod \"e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:34.822331 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.820643 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istiod-ca-cert\") pod \"e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:34.822331 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.820702 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istio-envoy\") pod \"e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:34.822331 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.820723 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-credential-socket\") pod \"e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:34.822331 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.820754 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istio-podinfo\") pod \"e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:34.822331 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.820877 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-workload-socket\") pod \"e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:34.822331 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.820914 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc2bt\" (UniqueName: \"kubernetes.io/projected/5669c7e0-fefa-4212-b3c9-411e65c59fa5-kube-api-access-cc2bt\") pod \"e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:34.822331 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.820939 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-workload-certs\") pod \"e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:34.822331 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.821032 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istio-token\") pod \"e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:34.923102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.922980 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-workload-socket\") pod \"e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:34.923102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.923057 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc2bt\" (UniqueName: \"kubernetes.io/projected/5669c7e0-fefa-4212-b3c9-411e65c59fa5-kube-api-access-cc2bt\") pod \"e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:34.923376 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.923104 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-workload-certs\") pod \"e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:34.923376 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.923178 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istio-token\") pod \"e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:34.923376 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.923285 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istio-data\") pod \"e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:34.923376 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.923351 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istiod-ca-cert\") pod \"e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:34.923874 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.923424 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istio-envoy\") pod \"e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:34.923874 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.923515 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-workload-socket\") pod \"e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:34.923874 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.923612 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-workload-certs\") pod \"e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:34.923874 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.923662 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-credential-socket\") pod \"e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:34.923874 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.923702 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istio-podinfo\") pod \"e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:34.924145 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.924120 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-credential-socket\") pod \"e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:34.924283 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.924266 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istio-data\") pod \"e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:34.924380 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.924349 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istiod-ca-cert\") pod \"e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:34.925938 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.925917 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istio-envoy\") pod \"e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:34.926205 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.926189 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istio-podinfo\") pod \"e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:34.981498 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.981464 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istio-token\") pod \"e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:34.981699 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:34.981673 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc2bt\" (UniqueName: \"kubernetes.io/projected/5669c7e0-fefa-4212-b3c9-411e65c59fa5-kube-api-access-cc2bt\") pod \"e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:35.274819 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:35.274763 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:35.707208 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:35.706167 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp"] Nov 03 21:42:35.748833 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:35.746483 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:42:35.748833 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:35.746607 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:42:35.748833 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:35.746667 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:42:35.838306 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:35.838259 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" event={"ID":"5669c7e0-fefa-4212-b3c9-411e65c59fa5","Type":"ContainerStarted","Data":"d253e56f6659469435634c16addadf68d0434f87ffbacd6d161a05db514ff3b2"} Nov 03 21:42:36.846177 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:36.846136 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" event={"ID":"5669c7e0-fefa-4212-b3c9-411e65c59fa5","Type":"ContainerStarted","Data":"32109f01797a2eb2b02f8d7e91348bfe43851b371acc790c9183e46ca5118667"} Nov 03 21:42:36.964384 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:36.964324 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" podStartSLOduration=2.964309336 podStartE2EDuration="2.964309336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:36.960733528 +0000 UTC m=+3251.428512248" watchObservedRunningTime="2026-11-03 21:42:36.964309336 +0000 UTC m=+3251.432088059" Nov 03 21:42:37.275689 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:37.275657 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:37.281870 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:37.281842 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:37.852594 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:37.852557 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:37.853732 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:37.853707 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:42:40.619393 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:42:40.619356 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:42:44.774915 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:44.774872 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5"] Nov 03 21:42:44.779533 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:44.779511 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:44.829314 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:44.829273 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-a3dc77-openshift-default-dockercfg-qtvdp\"" Nov 03 21:42:44.879442 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:44.879406 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5"] Nov 03 21:42:44.963309 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:44.963267 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-workload-certs\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:44.963488 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:44.963319 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93610f99-51b4-40f3-a2b2-19b628d9331f-istiod-ca-cert\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:44.963488 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:44.963454 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93610f99-51b4-40f3-a2b2-19b628d9331f-istio-token\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:44.963488 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:44.963479 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-istio-envoy\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:44.963601 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:44.963528 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93610f99-51b4-40f3-a2b2-19b628d9331f-istio-podinfo\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:44.963636 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:44.963591 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfkwt\" (UniqueName: \"kubernetes.io/projected/93610f99-51b4-40f3-a2b2-19b628d9331f-kube-api-access-gfkwt\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:44.963694 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:44.963672 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-credential-socket\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:44.963731 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:44.963716 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-workload-socket\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:44.963769 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:44.963750 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-istio-data\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:45.064846 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:45.064766 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-workload-certs\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:45.064846 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:45.064811 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93610f99-51b4-40f3-a2b2-19b628d9331f-istiod-ca-cert\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:45.065120 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:45.064874 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93610f99-51b4-40f3-a2b2-19b628d9331f-istio-token\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:45.065120 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:45.064892 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-istio-envoy\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:45.065120 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:45.064960 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93610f99-51b4-40f3-a2b2-19b628d9331f-istio-podinfo\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:45.065120 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:45.064989 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfkwt\" (UniqueName: \"kubernetes.io/projected/93610f99-51b4-40f3-a2b2-19b628d9331f-kube-api-access-gfkwt\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:45.065120 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:45.065026 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-credential-socket\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:45.065120 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:45.065064 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-workload-socket\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:45.065499 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:45.065130 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-istio-data\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:45.065574 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:45.065486 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-workload-certs\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:45.065574 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:45.065489 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-credential-socket\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:45.065693 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:45.065603 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-workload-socket\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:45.065894 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:45.065867 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-istio-data\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:45.065973 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:45.065930 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93610f99-51b4-40f3-a2b2-19b628d9331f-istiod-ca-cert\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:45.067348 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:45.067325 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-istio-envoy\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:45.067683 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:45.067660 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93610f99-51b4-40f3-a2b2-19b628d9331f-istio-podinfo\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:45.123619 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:45.123581 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93610f99-51b4-40f3-a2b2-19b628d9331f-istio-token\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:45.123798 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:45.123650 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfkwt\" (UniqueName: \"kubernetes.io/projected/93610f99-51b4-40f3-a2b2-19b628d9331f-kube-api-access-gfkwt\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:45.398608 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:45.398517 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:45.880221 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:45.880194 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5"] Nov 03 21:42:45.882138 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:42:45.882116 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93610f99_51b4_40f3_a2b2_19b628d9331f.slice/crio-9e0c78640c7a092f1d3a9bd311c1e963f614e43b8c62f4377aabf997e57d75fa WatchSource:0}: Error finding container 9e0c78640c7a092f1d3a9bd311c1e963f614e43b8c62f4377aabf997e57d75fa: Status 404 returned error can't find the container with id 9e0c78640c7a092f1d3a9bd311c1e963f614e43b8c62f4377aabf997e57d75fa Nov 03 21:42:45.884761 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:45.884725 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:42:45.884834 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:45.884810 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:42:45.884883 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:45.884839 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:42:45.906360 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:45.906331 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" event={"ID":"93610f99-51b4-40f3-a2b2-19b628d9331f","Type":"ContainerStarted","Data":"9e0c78640c7a092f1d3a9bd311c1e963f614e43b8c62f4377aabf997e57d75fa"} Nov 03 21:42:46.928340 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:46.928296 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" event={"ID":"93610f99-51b4-40f3-a2b2-19b628d9331f","Type":"ContainerStarted","Data":"5849084697f08275c04d514e84e7b404822b6ce6a13352d6bcc2402b0bbd6f73"} Nov 03 21:42:47.414099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:47.410908 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:47.414099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:47.411051 5615 patch_prober.go:29] interesting pod/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 03 21:42:47.414099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:47.411123 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" podUID="93610f99-51b4-40f3-a2b2-19b628d9331f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 03 21:42:48.411098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:48.406498 5615 patch_prober.go:29] interesting pod/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 03 21:42:48.411098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:48.406558 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" podUID="93610f99-51b4-40f3-a2b2-19b628d9331f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 03 21:42:49.422783 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:49.422520 5615 patch_prober.go:29] interesting pod/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 03 21:42:49.422783 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:49.422585 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" podUID="93610f99-51b4-40f3-a2b2-19b628d9331f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 03 21:42:50.418240 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:50.418195 5615 patch_prober.go:29] interesting pod/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 03 21:42:50.418655 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:50.418619 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" podUID="93610f99-51b4-40f3-a2b2-19b628d9331f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 03 21:42:51.403103 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:51.402524 5615 patch_prober.go:29] interesting pod/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 03 21:42:51.403103 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:51.402595 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" podUID="93610f99-51b4-40f3-a2b2-19b628d9331f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 03 21:42:52.404017 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:52.403984 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:52.404497 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:52.404335 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:52.405048 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:52.405029 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:42:52.569864 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:42:52.569803 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" podStartSLOduration=8.569787141 podStartE2EDuration="8.569787141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:47.091131334 +0000 UTC m=+3261.558910057" watchObservedRunningTime="2026-11-03 21:42:52.569787141 +0000 UTC m=+3267.037565864" Nov 03 21:43:10.620449 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:43:10.620403 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:43:40.619450 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:43:40.619411 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:44:10.619555 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:44:10.619464 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:44:33.752249 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:33.752200 5615 patch_prober.go:29] interesting pod/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:44:33.752962 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:33.752926 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" podUID="5b1a343a-cae6-4551-b189-bb6b0e978ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:44:34.824504 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:34.824411 5615 generic.go:356] "Generic (PLEG): container finished" podID="5669c7e0-fefa-4212-b3c9-411e65c59fa5" containerID="32109f01797a2eb2b02f8d7e91348bfe43851b371acc790c9183e46ca5118667" exitCode=0 Nov 03 21:44:34.824504 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:34.824484 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" event={"ID":"5669c7e0-fefa-4212-b3c9-411e65c59fa5","Type":"ContainerDied","Data":"32109f01797a2eb2b02f8d7e91348bfe43851b371acc790c9183e46ca5118667"} Nov 03 21:44:34.825172 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:34.824888 5615 scope.go:122] "RemoveContainer" containerID="32109f01797a2eb2b02f8d7e91348bfe43851b371acc790c9183e46ca5118667" Nov 03 21:44:34.826567 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:34.826534 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:44:34.826678 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:34.826606 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:44:34.826678 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:34.826638 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:44:35.832486 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:35.832453 5615 generic.go:356] "Generic (PLEG): container finished" podID="5b1a343a-cae6-4551-b189-bb6b0e978ba4" containerID="fc0200cb8c2785af885a6f131d47cfe4d0892b1f1850a60b68dcfcaf6e05cad5" exitCode=0 Nov 03 21:44:35.832994 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:35.832531 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" event={"ID":"5b1a343a-cae6-4551-b189-bb6b0e978ba4","Type":"ContainerDied","Data":"fc0200cb8c2785af885a6f131d47cfe4d0892b1f1850a60b68dcfcaf6e05cad5"} Nov 03 21:44:35.832994 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:35.832971 5615 scope.go:122] "RemoveContainer" containerID="fc0200cb8c2785af885a6f131d47cfe4d0892b1f1850a60b68dcfcaf6e05cad5" Nov 03 21:44:35.834773 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:35.834742 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:44:35.834873 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:35.834758 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" event={"ID":"5669c7e0-fefa-4212-b3c9-411e65c59fa5","Type":"ContainerStarted","Data":"8603bb3f51e339bda47af34d6f43ece73cf6fcc0d3063d59faaf1a34f8c664f7"} Nov 03 21:44:35.834873 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:35.834830 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:44:35.834965 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:35.834873 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:44:36.274998 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:36.274958 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:44:36.276220 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:36.276190 5615 patch_prober.go:29] interesting pod/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 21:44:36.276334 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:36.276254 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" podUID="5669c7e0-fefa-4212-b3c9-411e65c59fa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 21:44:36.851346 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:36.851306 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" event={"ID":"5b1a343a-cae6-4551-b189-bb6b0e978ba4","Type":"ContainerStarted","Data":"993f796158aadb4dc3d38ff297d56f07862f970abfe91782720ee107fa388f84"} Nov 03 21:44:37.275737 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:37.275703 5615 patch_prober.go:29] interesting pod/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 21:44:37.275987 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:37.275968 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" podUID="5669c7e0-fefa-4212-b3c9-411e65c59fa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 21:44:37.822102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:37.821553 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:44:37.822320 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:37.822178 5615 patch_prober.go:29] interesting pod/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 03 21:44:37.822320 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:37.822235 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" podUID="5b1a343a-cae6-4551-b189-bb6b0e978ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 03 21:44:37.852605 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:37.852573 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:44:38.275553 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:38.275515 5615 patch_prober.go:29] interesting pod/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 21:44:38.276253 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:38.276049 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" podUID="5669c7e0-fefa-4212-b3c9-411e65c59fa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 21:44:38.819459 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:38.819417 5615 patch_prober.go:29] interesting pod/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 03 21:44:38.819811 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:38.819771 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" podUID="5b1a343a-cae6-4551-b189-bb6b0e978ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 03 21:44:39.275377 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:39.275342 5615 patch_prober.go:29] interesting pod/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 21:44:39.275867 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:39.275843 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" podUID="5669c7e0-fefa-4212-b3c9-411e65c59fa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 21:44:39.820100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:39.819508 5615 patch_prober.go:29] interesting pod/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 03 21:44:39.820100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:39.819575 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" podUID="5b1a343a-cae6-4551-b189-bb6b0e978ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 03 21:44:40.275464 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:40.275421 5615 patch_prober.go:29] interesting pod/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 21:44:40.275882 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:40.275483 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" podUID="5669c7e0-fefa-4212-b3c9-411e65c59fa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 21:44:40.667657 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:44:40.667571 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:44:40.820101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:40.819581 5615 patch_prober.go:29] interesting pod/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 03 21:44:40.820101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:40.819651 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" podUID="5b1a343a-cae6-4551-b189-bb6b0e978ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 03 21:44:41.281119 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:41.281070 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:44:41.282059 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:41.282035 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:44:41.820664 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:41.820625 5615 patch_prober.go:29] interesting pod/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 03 21:44:41.820949 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:41.820918 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" podUID="5b1a343a-cae6-4551-b189-bb6b0e978ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 03 21:44:42.823503 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:42.823470 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:44:42.823921 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:42.823901 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:44:42.824676 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:44:42.824655 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:45:00.239187 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:00.239147 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm"] Nov 03 21:45:00.243015 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:00.242986 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj"] Nov 03 21:45:00.243224 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:00.243197 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm" Nov 03 21:45:00.246919 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:00.246898 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj" Nov 03 21:45:00.277755 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:00.277718 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/302f7223-1e4f-4615-8877-01e94b6a95c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj\" (UID: \"302f7223-1e4f-4615-8877-01e94b6a95c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj" Nov 03 21:45:00.277916 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:00.277773 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvs7d\" (UniqueName: \"kubernetes.io/projected/302f7223-1e4f-4615-8877-01e94b6a95c5-kube-api-access-dvs7d\") pod \"maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj\" (UID: \"302f7223-1e4f-4615-8877-01e94b6a95c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj" Nov 03 21:45:00.277916 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:00.277856 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4hnj\" (UniqueName: \"kubernetes.io/projected/91b7e5aa-2430-425f-86c7-b06c0dd0807b-kube-api-access-w4hnj\") pod \"maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm\" (UID: \"91b7e5aa-2430-425f-86c7-b06c0dd0807b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm" Nov 03 21:45:00.278018 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:00.277988 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91b7e5aa-2430-425f-86c7-b06c0dd0807b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm\" (UID: \"91b7e5aa-2430-425f-86c7-b06c0dd0807b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm" Nov 03 21:45:00.323731 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:00.323697 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\"" Nov 03 21:45:00.350285 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:00.350201 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm"] Nov 03 21:45:00.354902 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:00.354804 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj"] Nov 03 21:45:00.379745 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:00.379704 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91b7e5aa-2430-425f-86c7-b06c0dd0807b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm\" (UID: \"91b7e5aa-2430-425f-86c7-b06c0dd0807b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm" Nov 03 21:45:00.379940 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:00.379856 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/302f7223-1e4f-4615-8877-01e94b6a95c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj\" (UID: \"302f7223-1e4f-4615-8877-01e94b6a95c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj" Nov 03 21:45:00.379940 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:00.379901 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvs7d\" (UniqueName: \"kubernetes.io/projected/302f7223-1e4f-4615-8877-01e94b6a95c5-kube-api-access-dvs7d\") pod \"maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj\" (UID: \"302f7223-1e4f-4615-8877-01e94b6a95c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj" Nov 03 21:45:00.380118 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:00.379937 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4hnj\" (UniqueName: \"kubernetes.io/projected/91b7e5aa-2430-425f-86c7-b06c0dd0807b-kube-api-access-w4hnj\") pod \"maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm\" (UID: \"91b7e5aa-2430-425f-86c7-b06c0dd0807b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm" Nov 03 21:45:00.380454 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:00.380429 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91b7e5aa-2430-425f-86c7-b06c0dd0807b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm\" (UID: \"91b7e5aa-2430-425f-86c7-b06c0dd0807b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm" Nov 03 21:45:00.381025 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:00.380999 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/302f7223-1e4f-4615-8877-01e94b6a95c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj\" (UID: \"302f7223-1e4f-4615-8877-01e94b6a95c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj" Nov 03 21:45:00.439829 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:00.439796 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvs7d\" (UniqueName: \"kubernetes.io/projected/302f7223-1e4f-4615-8877-01e94b6a95c5-kube-api-access-dvs7d\") pod \"maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj\" (UID: \"302f7223-1e4f-4615-8877-01e94b6a95c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj" Nov 03 21:45:00.439982 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:00.439794 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4hnj\" (UniqueName: \"kubernetes.io/projected/91b7e5aa-2430-425f-86c7-b06c0dd0807b-kube-api-access-w4hnj\") pod \"maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm\" (UID: \"91b7e5aa-2430-425f-86c7-b06c0dd0807b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm" Nov 03 21:45:00.560330 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:00.560226 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm" Nov 03 21:45:00.566622 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:00.566593 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj" Nov 03 21:45:00.930557 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:00.930516 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm"] Nov 03 21:45:00.932841 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:45:00.932809 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b7e5aa_2430_425f_86c7_b06c0dd0807b.slice/crio-bce011a48fd1e437d3779b5ea8cb4787581b427ebf9b8cc644de05e435ccb856 WatchSource:0}: Error finding container bce011a48fd1e437d3779b5ea8cb4787581b427ebf9b8cc644de05e435ccb856: Status 404 returned error can't find the container with id bce011a48fd1e437d3779b5ea8cb4787581b427ebf9b8cc644de05e435ccb856 Nov 03 21:45:00.979117 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:00.979090 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj"] Nov 03 21:45:01.005008 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:01.004972 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm" event={"ID":"91b7e5aa-2430-425f-86c7-b06c0dd0807b","Type":"ContainerStarted","Data":"bce011a48fd1e437d3779b5ea8cb4787581b427ebf9b8cc644de05e435ccb856"} Nov 03 21:45:01.006338 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:01.006313 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj" event={"ID":"302f7223-1e4f-4615-8877-01e94b6a95c5","Type":"ContainerStarted","Data":"5537509ea8bb60a53a5cde1c44d7851b935545415326caee78f1331e4cbce733"} Nov 03 21:45:02.015720 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:02.015688 5615 generic.go:356] "Generic (PLEG): container finished" podID="302f7223-1e4f-4615-8877-01e94b6a95c5" containerID="43e87b2df6d40f10a327008ba1be60a1f26a6a0c418c9b585e8fc5d7ec98d55a" exitCode=60 Nov 03 21:45:02.016126 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:02.015769 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj" event={"ID":"302f7223-1e4f-4615-8877-01e94b6a95c5","Type":"ContainerDied","Data":"43e87b2df6d40f10a327008ba1be60a1f26a6a0c418c9b585e8fc5d7ec98d55a"} Nov 03 21:45:02.016126 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:02.016062 5615 scope.go:122] "RemoveContainer" containerID="43e87b2df6d40f10a327008ba1be60a1f26a6a0c418c9b585e8fc5d7ec98d55a" Nov 03 21:45:02.017768 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:02.017748 5615 generic.go:356] "Generic (PLEG): container finished" podID="91b7e5aa-2430-425f-86c7-b06c0dd0807b" containerID="b4bc5e5fc69100da4587939cae507592b3cf916a023b4d0cf05b9bbc7e2b843a" exitCode=60 Nov 03 21:45:02.017855 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:02.017804 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm" event={"ID":"91b7e5aa-2430-425f-86c7-b06c0dd0807b","Type":"ContainerDied","Data":"b4bc5e5fc69100da4587939cae507592b3cf916a023b4d0cf05b9bbc7e2b843a"} Nov 03 21:45:02.018127 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:02.018110 5615 scope.go:122] "RemoveContainer" containerID="b4bc5e5fc69100da4587939cae507592b3cf916a023b4d0cf05b9bbc7e2b843a" Nov 03 21:45:03.027559 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:03.027523 5615 generic.go:356] "Generic (PLEG): container finished" podID="302f7223-1e4f-4615-8877-01e94b6a95c5" containerID="5f524a3bf6a746c8f4d9e7f8a915dc7c8dd8c7f681f4ba782bd0a01913a13b38" exitCode=60 Nov 03 21:45:03.028019 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:03.027606 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj" event={"ID":"302f7223-1e4f-4615-8877-01e94b6a95c5","Type":"ContainerDied","Data":"5f524a3bf6a746c8f4d9e7f8a915dc7c8dd8c7f681f4ba782bd0a01913a13b38"} Nov 03 21:45:03.028019 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:03.027680 5615 scope.go:122] "RemoveContainer" containerID="43e87b2df6d40f10a327008ba1be60a1f26a6a0c418c9b585e8fc5d7ec98d55a" Nov 03 21:45:03.028019 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:03.027998 5615 scope.go:122] "RemoveContainer" containerID="5f524a3bf6a746c8f4d9e7f8a915dc7c8dd8c7f681f4ba782bd0a01913a13b38" Nov 03 21:45:03.028372 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:45:03.028348 5615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj_odh-ai-gateway-infra(302f7223-1e4f-4615-8877-01e94b6a95c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj" podUID="302f7223-1e4f-4615-8877-01e94b6a95c5" Nov 03 21:45:03.029967 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:03.029944 5615 generic.go:356] "Generic (PLEG): container finished" podID="91b7e5aa-2430-425f-86c7-b06c0dd0807b" containerID="3858a69f6bd22c2e0fa5902e1238590899781df2e276455d34e5fc7ac502621a" exitCode=60 Nov 03 21:45:03.030174 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:03.029976 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm" event={"ID":"91b7e5aa-2430-425f-86c7-b06c0dd0807b","Type":"ContainerDied","Data":"3858a69f6bd22c2e0fa5902e1238590899781df2e276455d34e5fc7ac502621a"} Nov 03 21:45:03.030394 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:03.030378 5615 scope.go:122] "RemoveContainer" containerID="3858a69f6bd22c2e0fa5902e1238590899781df2e276455d34e5fc7ac502621a" Nov 03 21:45:03.030678 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:45:03.030658 5615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm_odh-ai-gateway-infra(91b7e5aa-2430-425f-86c7-b06c0dd0807b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm" podUID="91b7e5aa-2430-425f-86c7-b06c0dd0807b" Nov 03 21:45:03.071014 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:03.070990 5615 scope.go:122] "RemoveContainer" containerID="b4bc5e5fc69100da4587939cae507592b3cf916a023b4d0cf05b9bbc7e2b843a" Nov 03 21:45:04.039793 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:04.039767 5615 scope.go:122] "RemoveContainer" containerID="5f524a3bf6a746c8f4d9e7f8a915dc7c8dd8c7f681f4ba782bd0a01913a13b38" Nov 03 21:45:04.040308 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:45:04.040096 5615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj_odh-ai-gateway-infra(302f7223-1e4f-4615-8877-01e94b6a95c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj" podUID="302f7223-1e4f-4615-8877-01e94b6a95c5" Nov 03 21:45:04.041765 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:04.041745 5615 scope.go:122] "RemoveContainer" containerID="3858a69f6bd22c2e0fa5902e1238590899781df2e276455d34e5fc7ac502621a" Nov 03 21:45:04.042036 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:45:04.042016 5615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm_odh-ai-gateway-infra(91b7e5aa-2430-425f-86c7-b06c0dd0807b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm" podUID="91b7e5aa-2430-425f-86c7-b06c0dd0807b" Nov 03 21:45:08.916158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:08.916112 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76"] Nov 03 21:45:08.920598 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:08.920572 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:08.988095 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:08.981789 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-6a186d-openshift-default-dockercfg-ztlm7\"" Nov 03 21:45:09.037520 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.037331 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-istio-envoy\") pod \"e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:09.039645 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.039616 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llj4x\" (UniqueName: \"kubernetes.io/projected/4615ffcf-3653-436d-bfc4-ecda29430be4-kube-api-access-llj4x\") pod \"e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:09.039916 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.039896 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-credential-socket\") pod \"e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:09.040429 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.040381 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76"] Nov 03 21:45:09.040971 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.040945 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-workload-certs\") pod \"e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:09.041157 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.041129 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4615ffcf-3653-436d-bfc4-ecda29430be4-istio-podinfo\") pod \"e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:09.044107 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.044060 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4615ffcf-3653-436d-bfc4-ecda29430be4-istio-token\") pod \"e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:09.050694 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.050660 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-workload-socket\") pod \"e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:09.050936 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.050919 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4615ffcf-3653-436d-bfc4-ecda29430be4-istiod-ca-cert\") pod \"e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:09.051062 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.051047 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-istio-data\") pod \"e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:09.153700 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.152135 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-workload-socket\") pod \"e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:09.153700 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.152188 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4615ffcf-3653-436d-bfc4-ecda29430be4-istiod-ca-cert\") pod \"e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:09.153700 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.152225 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-istio-data\") pod \"e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:09.153700 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.152277 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-istio-envoy\") pod \"e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:09.153700 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.152373 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llj4x\" (UniqueName: \"kubernetes.io/projected/4615ffcf-3653-436d-bfc4-ecda29430be4-kube-api-access-llj4x\") pod \"e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:09.153700 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.152448 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-credential-socket\") pod \"e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:09.153700 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.152562 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-workload-certs\") pod \"e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:09.153700 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.152601 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4615ffcf-3653-436d-bfc4-ecda29430be4-istio-podinfo\") pod \"e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:09.153700 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.152642 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4615ffcf-3653-436d-bfc4-ecda29430be4-istio-token\") pod \"e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:09.153700 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.153046 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4615ffcf-3653-436d-bfc4-ecda29430be4-istiod-ca-cert\") pod \"e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:09.153700 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.153384 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-istio-data\") pod \"e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:09.153700 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.153424 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-credential-socket\") pod \"e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:09.153700 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.153660 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-workload-certs\") pod \"e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:09.154367 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.154106 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-workload-socket\") pod \"e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:09.156410 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.156377 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4615ffcf-3653-436d-bfc4-ecda29430be4-istio-podinfo\") pod \"e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:09.156955 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.156934 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-istio-envoy\") pod \"e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:09.213759 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.213677 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4615ffcf-3653-436d-bfc4-ecda29430be4-istio-token\") pod \"e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:09.216837 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.216801 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llj4x\" (UniqueName: \"kubernetes.io/projected/4615ffcf-3653-436d-bfc4-ecda29430be4-kube-api-access-llj4x\") pod \"e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:09.235830 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.235802 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:09.657161 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.657133 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76"] Nov 03 21:45:09.659138 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:45:09.659104 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4615ffcf_3653_436d_bfc4_ecda29430be4.slice/crio-3921dd03d0276af4e5ac301d1696a4d2dc61d79a2f1e1cf1468d809ed9110961 WatchSource:0}: Error finding container 3921dd03d0276af4e5ac301d1696a4d2dc61d79a2f1e1cf1468d809ed9110961: Status 404 returned error can't find the container with id 3921dd03d0276af4e5ac301d1696a4d2dc61d79a2f1e1cf1468d809ed9110961 Nov 03 21:45:09.661703 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.661666 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:45:09.661775 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.661738 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:45:09.661775 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:09.661767 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:45:10.156351 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:10.156312 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" event={"ID":"4615ffcf-3653-436d-bfc4-ecda29430be4","Type":"ContainerStarted","Data":"afa3e57ac5ee5015b83f3fa5cfa7dd3243592976bed789f6ae5eebc2c7787bf1"} Nov 03 21:45:10.156762 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:10.156455 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" event={"ID":"4615ffcf-3653-436d-bfc4-ecda29430be4","Type":"ContainerStarted","Data":"3921dd03d0276af4e5ac301d1696a4d2dc61d79a2f1e1cf1468d809ed9110961"} Nov 03 21:45:10.241429 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:10.241379 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:10.257223 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:10.257181 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:10.272837 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:10.272784 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" podStartSLOduration=2.272770137 podStartE2EDuration="2.272770137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:10.271576005 +0000 UTC m=+3404.739354729" watchObservedRunningTime="2026-11-03 21:45:10.272770137 +0000 UTC m=+3404.740548860" Nov 03 21:45:10.619254 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:45:10.619214 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:45:11.163020 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:11.162986 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:11.163952 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:11.163933 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:45:14.645180 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:14.645141 5615 scope.go:122] "RemoveContainer" containerID="5f524a3bf6a746c8f4d9e7f8a915dc7c8dd8c7f681f4ba782bd0a01913a13b38" Nov 03 21:45:15.189421 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:15.189326 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj" event={"ID":"302f7223-1e4f-4615-8877-01e94b6a95c5","Type":"ContainerStarted","Data":"80ebfe9e57f3f9fd499e4ff6c17b252632321e459944762752c4338f096623e9"} Nov 03 21:45:15.301162 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:15.301102 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj" podStartSLOduration=15.301088613 podStartE2EDuration="15.301088613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:15.299574962 +0000 UTC m=+3409.767353689" watchObservedRunningTime="2026-11-03 21:45:15.301088613 +0000 UTC m=+3409.768867335" Nov 03 21:45:16.204893 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:16.204845 5615 generic.go:356] "Generic (PLEG): container finished" podID="302f7223-1e4f-4615-8877-01e94b6a95c5" containerID="80ebfe9e57f3f9fd499e4ff6c17b252632321e459944762752c4338f096623e9" exitCode=60 Nov 03 21:45:16.205436 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:16.204901 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj" event={"ID":"302f7223-1e4f-4615-8877-01e94b6a95c5","Type":"ContainerDied","Data":"80ebfe9e57f3f9fd499e4ff6c17b252632321e459944762752c4338f096623e9"} Nov 03 21:45:16.205436 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:16.205220 5615 scope.go:122] "RemoveContainer" containerID="5f524a3bf6a746c8f4d9e7f8a915dc7c8dd8c7f681f4ba782bd0a01913a13b38" Nov 03 21:45:16.206473 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:16.206451 5615 scope.go:122] "RemoveContainer" containerID="80ebfe9e57f3f9fd499e4ff6c17b252632321e459944762752c4338f096623e9" Nov 03 21:45:16.206927 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:45:16.206860 5615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj_odh-ai-gateway-infra(302f7223-1e4f-4615-8877-01e94b6a95c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj" podUID="302f7223-1e4f-4615-8877-01e94b6a95c5" Nov 03 21:45:16.311602 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:16.311559 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj"] Nov 03 21:45:17.385096 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:17.385044 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj" Nov 03 21:45:17.472413 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:17.472304 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/302f7223-1e4f-4615-8877-01e94b6a95c5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/302f7223-1e4f-4615-8877-01e94b6a95c5-maas-api-service-ca\") pod \"302f7223-1e4f-4615-8877-01e94b6a95c5\" (UID: \"302f7223-1e4f-4615-8877-01e94b6a95c5\") " Nov 03 21:45:17.472584 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:17.472444 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/302f7223-1e4f-4615-8877-01e94b6a95c5-kube-api-access-dvs7d\" (UniqueName: \"kubernetes.io/projected/302f7223-1e4f-4615-8877-01e94b6a95c5-kube-api-access-dvs7d\") pod \"302f7223-1e4f-4615-8877-01e94b6a95c5\" (UID: \"302f7223-1e4f-4615-8877-01e94b6a95c5\") " Nov 03 21:45:17.472816 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:17.472788 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/302f7223-1e4f-4615-8877-01e94b6a95c5-maas-api-service-ca" pod "302f7223-1e4f-4615-8877-01e94b6a95c5" (UID: "302f7223-1e4f-4615-8877-01e94b6a95c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:45:17.473056 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:17.473042 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/302f7223-1e4f-4615-8877-01e94b6a95c5-maas-api-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:45:17.474596 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:17.474573 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/302f7223-1e4f-4615-8877-01e94b6a95c5-kube-api-access-dvs7d" pod "302f7223-1e4f-4615-8877-01e94b6a95c5" (UID: "302f7223-1e4f-4615-8877-01e94b6a95c5"). InnerVolumeSpecName "kube-api-access-dvs7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:45:17.574180 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:17.574136 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvs7d\" (UniqueName: \"kubernetes.io/projected/302f7223-1e4f-4615-8877-01e94b6a95c5-kube-api-access-dvs7d\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:45:17.644981 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:17.644945 5615 scope.go:122] "RemoveContainer" containerID="3858a69f6bd22c2e0fa5902e1238590899781df2e276455d34e5fc7ac502621a" Nov 03 21:45:18.221245 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:18.221144 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm" event={"ID":"91b7e5aa-2430-425f-86c7-b06c0dd0807b","Type":"ContainerStarted","Data":"96cf52569ee678a19921eee5fb7329825c12165197e78018773cd8074810ce60"} Nov 03 21:45:18.222718 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:18.222695 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj" Nov 03 21:45:18.222851 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:18.222737 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj" event={"ID":"302f7223-1e4f-4615-8877-01e94b6a95c5","Type":"ContainerDied","Data":"5537509ea8bb60a53a5cde1c44d7851b935545415326caee78f1331e4cbce733"} Nov 03 21:45:18.222851 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:18.222806 5615 scope.go:122] "RemoveContainer" containerID="80ebfe9e57f3f9fd499e4ff6c17b252632321e459944762752c4338f096623e9" Nov 03 21:45:18.334744 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:18.334687 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm" podStartSLOduration=18.334673655 podStartE2EDuration="18.334673655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:18.331710128 +0000 UTC m=+3412.799488850" watchObservedRunningTime="2026-11-03 21:45:18.334673655 +0000 UTC m=+3412.802452378" Nov 03 21:45:18.491209 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:18.491173 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj"] Nov 03 21:45:18.511218 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:18.495188 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2057ca-29895705-vvbkj"] Nov 03 21:45:18.883155 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:18.882711 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6a186d-555fb5d966-t8ppx"] Nov 03 21:45:18.909100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:18.906262 5615 cpu_manager.go:414] "containerMap: removing container" podUID="302f7223-1e4f-4615-8877-01e94b6a95c5" containerName="cleanup" Nov 03 21:45:18.909100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:18.906298 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="302f7223-1e4f-4615-8877-01e94b6a95c5" containerName="cleanup" Nov 03 21:45:18.909100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:18.906352 5615 cpu_manager.go:414] "containerMap: removing container" podUID="302f7223-1e4f-4615-8877-01e94b6a95c5" containerName="cleanup" Nov 03 21:45:18.909100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:18.906360 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="302f7223-1e4f-4615-8877-01e94b6a95c5" containerName="cleanup" Nov 03 21:45:18.909100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:18.906391 5615 cpu_manager.go:414] "containerMap: removing container" podUID="302f7223-1e4f-4615-8877-01e94b6a95c5" containerName="cleanup" Nov 03 21:45:18.909100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:18.906399 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="302f7223-1e4f-4615-8877-01e94b6a95c5" containerName="cleanup" Nov 03 21:45:18.909100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:18.906590 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="302f7223-1e4f-4615-8877-01e94b6a95c5" containerName="cleanup" Nov 03 21:45:18.909100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:18.906605 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="302f7223-1e4f-4615-8877-01e94b6a95c5" containerName="cleanup" Nov 03 21:45:18.918672 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:18.910189 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6a186d-555fb5d966-t8ppx" Nov 03 21:45:18.955331 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:18.947532 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6a186d-555fb5d966-t8ppx"] Nov 03 21:45:18.965401 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:18.965183 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-6a186d-dockercfg-wlj9l\"" Nov 03 21:45:18.966288 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:18.966261 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-6a186d\"" Nov 03 21:45:19.052800 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:19.052764 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12271b60-16d6-4716-8db7-fddd1d5bbd8b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6a186d-555fb5d966-t8ppx\" (UID: \"12271b60-16d6-4716-8db7-fddd1d5bbd8b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6a186d-555fb5d966-t8ppx" Nov 03 21:45:19.052962 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:19.052942 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgj7q\" (UniqueName: \"kubernetes.io/projected/12271b60-16d6-4716-8db7-fddd1d5bbd8b-kube-api-access-wgj7q\") pod \"payload-processing-e2e-extmod-b-6a186d-555fb5d966-t8ppx\" (UID: \"12271b60-16d6-4716-8db7-fddd1d5bbd8b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6a186d-555fb5d966-t8ppx" Nov 03 21:45:19.153901 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:19.153807 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgj7q\" (UniqueName: \"kubernetes.io/projected/12271b60-16d6-4716-8db7-fddd1d5bbd8b-kube-api-access-wgj7q\") pod \"payload-processing-e2e-extmod-b-6a186d-555fb5d966-t8ppx\" (UID: \"12271b60-16d6-4716-8db7-fddd1d5bbd8b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6a186d-555fb5d966-t8ppx" Nov 03 21:45:19.154065 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:19.154005 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12271b60-16d6-4716-8db7-fddd1d5bbd8b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6a186d-555fb5d966-t8ppx\" (UID: \"12271b60-16d6-4716-8db7-fddd1d5bbd8b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6a186d-555fb5d966-t8ppx" Nov 03 21:45:19.154710 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:19.154681 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12271b60-16d6-4716-8db7-fddd1d5bbd8b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6a186d-555fb5d966-t8ppx\" (UID: \"12271b60-16d6-4716-8db7-fddd1d5bbd8b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6a186d-555fb5d966-t8ppx" Nov 03 21:45:19.213854 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:19.213818 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgj7q\" (UniqueName: \"kubernetes.io/projected/12271b60-16d6-4716-8db7-fddd1d5bbd8b-kube-api-access-wgj7q\") pod \"payload-processing-e2e-extmod-b-6a186d-555fb5d966-t8ppx\" (UID: \"12271b60-16d6-4716-8db7-fddd1d5bbd8b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6a186d-555fb5d966-t8ppx" Nov 03 21:45:19.232344 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:19.232312 5615 generic.go:356] "Generic (PLEG): container finished" podID="91b7e5aa-2430-425f-86c7-b06c0dd0807b" containerID="96cf52569ee678a19921eee5fb7329825c12165197e78018773cd8074810ce60" exitCode=60 Nov 03 21:45:19.232508 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:19.232386 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm" event={"ID":"91b7e5aa-2430-425f-86c7-b06c0dd0807b","Type":"ContainerDied","Data":"96cf52569ee678a19921eee5fb7329825c12165197e78018773cd8074810ce60"} Nov 03 21:45:19.232508 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:19.232446 5615 scope.go:122] "RemoveContainer" containerID="3858a69f6bd22c2e0fa5902e1238590899781df2e276455d34e5fc7ac502621a" Nov 03 21:45:19.232859 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:19.232841 5615 scope.go:122] "RemoveContainer" containerID="96cf52569ee678a19921eee5fb7329825c12165197e78018773cd8074810ce60" Nov 03 21:45:19.233261 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:45:19.233233 5615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm_odh-ai-gateway-infra(91b7e5aa-2430-425f-86c7-b06c0dd0807b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm" podUID="91b7e5aa-2430-425f-86c7-b06c0dd0807b" Nov 03 21:45:19.287644 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:19.287609 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6a186d-555fb5d966-t8ppx" Nov 03 21:45:19.344239 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:19.344114 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm"] Nov 03 21:45:19.653839 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:19.653792 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="302f7223-1e4f-4615-8877-01e94b6a95c5" path="/var/lib/kubelet/pods/302f7223-1e4f-4615-8877-01e94b6a95c5/volumes" Nov 03 21:45:19.661340 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:19.661310 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6a186d-555fb5d966-t8ppx"] Nov 03 21:45:19.662777 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:45:19.662747 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12271b60_16d6_4716_8db7_fddd1d5bbd8b.slice/crio-e500aa193b12c9f53def28f803978e02b2be07fe2aedf917f5a5666ea345edb6 WatchSource:0}: Error finding container e500aa193b12c9f53def28f803978e02b2be07fe2aedf917f5a5666ea345edb6: Status 404 returned error can't find the container with id e500aa193b12c9f53def28f803978e02b2be07fe2aedf917f5a5666ea345edb6 Nov 03 21:45:20.243293 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:20.243257 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6a186d-555fb5d966-t8ppx" event={"ID":"12271b60-16d6-4716-8db7-fddd1d5bbd8b","Type":"ContainerStarted","Data":"5594f4c1c6437d6669f454f97e9420e0f52e8024e631cd6c9328fb10a59ee83d"} Nov 03 21:45:20.243424 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:20.243310 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6a186d-555fb5d966-t8ppx" event={"ID":"12271b60-16d6-4716-8db7-fddd1d5bbd8b","Type":"ContainerStarted","Data":"e500aa193b12c9f53def28f803978e02b2be07fe2aedf917f5a5666ea345edb6"} Nov 03 21:45:20.243424 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:20.243346 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-6a186d-555fb5d966-t8ppx" Nov 03 21:45:20.460164 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:20.460134 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm" Nov 03 21:45:20.465838 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:20.465779 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-6a186d-555fb5d966-t8ppx" podStartSLOduration=2.465760791 podStartE2EDuration="2.465760791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:20.463559234 +0000 UTC m=+3414.931337958" watchObservedRunningTime="2026-11-03 21:45:20.465760791 +0000 UTC m=+3414.933539515" Nov 03 21:45:20.481138 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:20.481107 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91b7e5aa-2430-425f-86c7-b06c0dd0807b-kube-api-access-w4hnj\" (UniqueName: \"kubernetes.io/projected/91b7e5aa-2430-425f-86c7-b06c0dd0807b-kube-api-access-w4hnj\") pod \"91b7e5aa-2430-425f-86c7-b06c0dd0807b\" (UID: \"91b7e5aa-2430-425f-86c7-b06c0dd0807b\") " Nov 03 21:45:20.481297 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:20.481209 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91b7e5aa-2430-425f-86c7-b06c0dd0807b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91b7e5aa-2430-425f-86c7-b06c0dd0807b-maas-api-service-ca\") pod \"91b7e5aa-2430-425f-86c7-b06c0dd0807b\" (UID: \"91b7e5aa-2430-425f-86c7-b06c0dd0807b\") " Nov 03 21:45:20.481595 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:20.481570 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91b7e5aa-2430-425f-86c7-b06c0dd0807b-maas-api-service-ca" pod "91b7e5aa-2430-425f-86c7-b06c0dd0807b" (UID: "91b7e5aa-2430-425f-86c7-b06c0dd0807b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:45:20.483406 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:20.483384 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b7e5aa-2430-425f-86c7-b06c0dd0807b-kube-api-access-w4hnj" pod "91b7e5aa-2430-425f-86c7-b06c0dd0807b" (UID: "91b7e5aa-2430-425f-86c7-b06c0dd0807b"). InnerVolumeSpecName "kube-api-access-w4hnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:45:20.603641 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:20.603602 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4hnj\" (UniqueName: \"kubernetes.io/projected/91b7e5aa-2430-425f-86c7-b06c0dd0807b-kube-api-access-w4hnj\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:45:20.604482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:20.604414 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91b7e5aa-2430-425f-86c7-b06c0dd0807b-maas-api-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:45:21.251664 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:21.251622 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm" event={"ID":"91b7e5aa-2430-425f-86c7-b06c0dd0807b","Type":"ContainerDied","Data":"bce011a48fd1e437d3779b5ea8cb4787581b427ebf9b8cc644de05e435ccb856"} Nov 03 21:45:21.252212 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:21.251696 5615 scope.go:122] "RemoveContainer" containerID="96cf52569ee678a19921eee5fb7329825c12165197e78018773cd8074810ce60" Nov 03 21:45:21.252212 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:21.251636 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm" Nov 03 21:45:21.411017 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:21.410980 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm"] Nov 03 21:45:21.458367 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:21.458324 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-130a8c-29895705-pzxmm"] Nov 03 21:45:21.652213 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:21.652139 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91b7e5aa-2430-425f-86c7-b06c0dd0807b" path="/var/lib/kubelet/pods/91b7e5aa-2430-425f-86c7-b06c0dd0807b/volumes" Nov 03 21:45:31.255527 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:45:31.255495 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-6a186d-555fb5d966-t8ppx" Nov 03 21:45:40.654180 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:45:40.654097 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:46:10.619549 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:46:10.619509 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:46:13.051388 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.051349 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6"] Nov 03 21:46:13.052028 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.052012 5615 cpu_manager.go:414] "containerMap: removing container" podUID="91b7e5aa-2430-425f-86c7-b06c0dd0807b" containerName="cleanup" Nov 03 21:46:13.052109 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.052029 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91b7e5aa-2430-425f-86c7-b06c0dd0807b" containerName="cleanup" Nov 03 21:46:13.052109 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.052039 5615 cpu_manager.go:414] "containerMap: removing container" podUID="91b7e5aa-2430-425f-86c7-b06c0dd0807b" containerName="cleanup" Nov 03 21:46:13.052109 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.052044 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91b7e5aa-2430-425f-86c7-b06c0dd0807b" containerName="cleanup" Nov 03 21:46:13.052212 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.052192 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="91b7e5aa-2430-425f-86c7-b06c0dd0807b" containerName="cleanup" Nov 03 21:46:13.052212 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.052202 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="302f7223-1e4f-4615-8877-01e94b6a95c5" containerName="cleanup" Nov 03 21:46:13.052279 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.052216 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="91b7e5aa-2430-425f-86c7-b06c0dd0807b" containerName="cleanup" Nov 03 21:46:13.052279 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.052221 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="91b7e5aa-2430-425f-86c7-b06c0dd0807b" containerName="cleanup" Nov 03 21:46:13.058644 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.058607 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:13.109838 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.109796 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-7da0f7-openshift-default-dockercfg-h89nd\"" Nov 03 21:46:13.160309 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.160275 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6"] Nov 03 21:46:13.171467 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.171429 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfvnp\" (UniqueName: \"kubernetes.io/projected/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-kube-api-access-cfvnp\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:13.171849 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.171800 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-credential-socket\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:13.171849 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.171847 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istio-podinfo\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:13.172060 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.172009 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-workload-certs\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:13.172060 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.172042 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-workload-socket\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:13.172260 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.172109 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istio-envoy\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:13.172260 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.172139 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istio-token\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:13.172260 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.172161 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istio-data\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:13.172260 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.172200 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istiod-ca-cert\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:13.273792 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.273755 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfvnp\" (UniqueName: \"kubernetes.io/projected/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-kube-api-access-cfvnp\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:13.273986 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.273876 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-credential-socket\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:13.273986 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.273910 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istio-podinfo\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:13.274147 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.273997 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-workload-certs\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:13.274147 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.274024 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-workload-socket\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:13.274147 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.274089 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istio-envoy\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:13.274147 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.274120 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istio-token\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:13.280391 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.274147 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istio-data\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:13.280391 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.274184 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istiod-ca-cert\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:13.280391 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.275042 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istiod-ca-cert\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:13.280391 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.275685 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-credential-socket\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:13.280391 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.277267 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-workload-certs\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:13.280391 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.279489 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istio-envoy\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:13.280391 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.279707 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istio-data\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:13.280391 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.279878 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istio-podinfo\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:13.280391 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.279933 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-workload-socket\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:13.336387 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.336311 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfvnp\" (UniqueName: \"kubernetes.io/projected/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-kube-api-access-cfvnp\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:13.336707 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.336687 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istio-token\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:13.377057 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.377005 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:13.803060 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.803025 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6"] Nov 03 21:46:13.804559 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:46:13.804531 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc56f893_acbf_4db9_9a61_b7c4c0df44a7.slice/crio-1fac1bfefa3875a58a3fe68e0791571bb6bacac777ae45d2d5f786d687cd6af8 WatchSource:0}: Error finding container 1fac1bfefa3875a58a3fe68e0791571bb6bacac777ae45d2d5f786d687cd6af8: Status 404 returned error can't find the container with id 1fac1bfefa3875a58a3fe68e0791571bb6bacac777ae45d2d5f786d687cd6af8 Nov 03 21:46:13.807677 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.807638 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:46:13.807770 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.807729 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:46:13.807811 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:13.807769 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:46:14.669159 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:14.669110 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" event={"ID":"cc56f893-acbf-4db9-9a61-b7c4c0df44a7","Type":"ContainerStarted","Data":"0bbff94ce9766fc230fc8ea0b02250f0605f789c8be263568aa3ec7603801b53"} Nov 03 21:46:14.669700 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:14.669188 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" event={"ID":"cc56f893-acbf-4db9-9a61-b7c4c0df44a7","Type":"ContainerStarted","Data":"1fac1bfefa3875a58a3fe68e0791571bb6bacac777ae45d2d5f786d687cd6af8"} Nov 03 21:46:14.833740 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:14.833681 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" podStartSLOduration=1.83366619 podStartE2EDuration="1.83366619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:14.831744183 +0000 UTC m=+3469.299522914" watchObservedRunningTime="2026-11-03 21:46:14.83366619 +0000 UTC m=+3469.301444910" Nov 03 21:46:15.378212 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:15.378180 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:15.379923 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:15.379899 5615 patch_prober.go:29] interesting pod/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 03 21:46:15.379999 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:15.379960 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" podUID="cc56f893-acbf-4db9-9a61-b7c4c0df44a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 03 21:46:16.377823 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:16.377789 5615 patch_prober.go:29] interesting pod/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 03 21:46:16.378290 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:16.377839 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" podUID="cc56f893-acbf-4db9-9a61-b7c4c0df44a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 03 21:46:17.377980 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:17.377941 5615 patch_prober.go:29] interesting pod/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 03 21:46:17.378418 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:17.378000 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" podUID="cc56f893-acbf-4db9-9a61-b7c4c0df44a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 03 21:46:18.377880 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:18.377841 5615 patch_prober.go:29] interesting pod/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 03 21:46:18.378066 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:18.377899 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" podUID="cc56f893-acbf-4db9-9a61-b7c4c0df44a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 03 21:46:19.377627 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:19.377588 5615 patch_prober.go:29] interesting pod/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 03 21:46:19.377917 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:19.377642 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" podUID="cc56f893-acbf-4db9-9a61-b7c4c0df44a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 03 21:46:20.382758 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:20.382728 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:20.383218 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:20.383195 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:20.383874 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:20.383856 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:46:40.619681 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:46:40.619641 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:46:43.314045 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.313998 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt"] Nov 03 21:46:43.323100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.315783 5615 cpu_manager.go:414] "containerMap: removing container" podUID="91b7e5aa-2430-425f-86c7-b06c0dd0807b" containerName="cleanup" Nov 03 21:46:43.339599 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.339566 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91b7e5aa-2430-425f-86c7-b06c0dd0807b" containerName="cleanup" Nov 03 21:46:43.349535 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.349503 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:43.369048 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.369009 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt"] Nov 03 21:46:43.456098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.455008 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-istio-envoy\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:43.456098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.455167 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa117f5-40a0-439f-8008-2264c4c8573c-istio-podinfo\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:43.456098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.455231 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa117f5-40a0-439f-8008-2264c4c8573c-istio-token\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:43.456098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.455271 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-workload-socket\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:43.456098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.455298 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-workload-certs\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:43.456098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.455355 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-istio-data\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:43.456098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.455426 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-credential-socket\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:43.456098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.455464 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa117f5-40a0-439f-8008-2264c4c8573c-istiod-ca-cert\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:43.456098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.455504 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl82b\" (UniqueName: \"kubernetes.io/projected/faa117f5-40a0-439f-8008-2264c4c8573c-kube-api-access-nl82b\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:43.557371 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.557330 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-istio-envoy\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:43.557611 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.557455 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa117f5-40a0-439f-8008-2264c4c8573c-istio-podinfo\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:43.557611 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.557504 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa117f5-40a0-439f-8008-2264c4c8573c-istio-token\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:43.557611 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.557534 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-workload-socket\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:43.557611 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.557565 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-workload-certs\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:43.557611 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.557607 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-istio-data\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:43.557907 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.557671 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-credential-socket\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:43.557907 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.557700 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa117f5-40a0-439f-8008-2264c4c8573c-istiod-ca-cert\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:43.557907 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.557737 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl82b\" (UniqueName: \"kubernetes.io/projected/faa117f5-40a0-439f-8008-2264c4c8573c-kube-api-access-nl82b\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:43.558068 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.557937 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-workload-socket\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:43.558299 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.558271 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-istio-data\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:43.558452 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.558341 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-workload-certs\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:43.558452 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.558369 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-credential-socket\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:43.558829 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.558800 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa117f5-40a0-439f-8008-2264c4c8573c-istiod-ca-cert\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:43.560276 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.560257 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-istio-envoy\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:43.560425 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.560406 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa117f5-40a0-439f-8008-2264c4c8573c-istio-podinfo\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:43.616353 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.616268 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa117f5-40a0-439f-8008-2264c4c8573c-istio-token\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:43.616522 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.616364 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl82b\" (UniqueName: \"kubernetes.io/projected/faa117f5-40a0-439f-8008-2264c4c8573c-kube-api-access-nl82b\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:43.665754 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:43.665715 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:44.094542 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:44.094512 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt"] Nov 03 21:46:44.095782 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:46:44.095754 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa117f5_40a0_439f_8008_2264c4c8573c.slice/crio-b9828a243b51b93e9b5aad68e2f969f9e636f90510417137e8d5bd2b8f271dbd WatchSource:0}: Error finding container b9828a243b51b93e9b5aad68e2f969f9e636f90510417137e8d5bd2b8f271dbd: Status 404 returned error can't find the container with id b9828a243b51b93e9b5aad68e2f969f9e636f90510417137e8d5bd2b8f271dbd Nov 03 21:46:44.098647 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:44.098611 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:46:44.098753 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:44.098680 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:46:44.098753 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:44.098709 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:46:44.931655 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:44.931606 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" event={"ID":"faa117f5-40a0-439f-8008-2264c4c8573c","Type":"ContainerStarted","Data":"7807dd9619c813a6a82cfb06efe0e6d32941e8996110e1b27c848b7e5925216e"} Nov 03 21:46:44.932065 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:44.931676 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" event={"ID":"faa117f5-40a0-439f-8008-2264c4c8573c","Type":"ContainerStarted","Data":"b9828a243b51b93e9b5aad68e2f969f9e636f90510417137e8d5bd2b8f271dbd"} Nov 03 21:46:45.049380 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:45.049303 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" podStartSLOduration=2.049284287 podStartE2EDuration="2.049284287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:45.04596231 +0000 UTC m=+3499.513741070" watchObservedRunningTime="2026-11-03 21:46:45.049284287 +0000 UTC m=+3499.517063013" Nov 03 21:46:45.665993 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:45.665958 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:45.667676 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:45.667645 5615 patch_prober.go:29] interesting pod/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 03 21:46:45.667802 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:45.667698 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" podUID="faa117f5-40a0-439f-8008-2264c4c8573c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 03 21:46:46.667682 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:46.667637 5615 patch_prober.go:29] interesting pod/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 03 21:46:46.668339 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:46.668292 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" podUID="faa117f5-40a0-439f-8008-2264c4c8573c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 03 21:46:47.666135 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:47.666093 5615 patch_prober.go:29] interesting pod/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 03 21:46:47.666322 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:47.666159 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" podUID="faa117f5-40a0-439f-8008-2264c4c8573c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 03 21:46:48.666600 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:48.666549 5615 patch_prober.go:29] interesting pod/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 03 21:46:48.666996 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:48.666627 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" podUID="faa117f5-40a0-439f-8008-2264c4c8573c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 03 21:46:49.666921 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:49.666882 5615 patch_prober.go:29] interesting pod/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 03 21:46:49.667376 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:49.666952 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" podUID="faa117f5-40a0-439f-8008-2264c4c8573c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 03 21:46:50.672575 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:50.672541 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:50.673130 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:50.672833 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:50.673615 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:50.673597 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:46:50.887460 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:50.887421 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w"] Nov 03 21:46:55.887967 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:55.887915 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w" podUID="d8f75aac-8457-4abc-ad9f-6cbb8f462f6b" containerName="payload-pre-processing" containerID="cri-o://13eec92495be8ebec49db0dcb7cefd6b775c9dbdb5a0119ef6e41571ab9eb627" gracePeriod=25 Nov 03 21:46:56.113006 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:56.112816 5615 generic.go:356] "Generic (PLEG): container finished" podID="d8f75aac-8457-4abc-ad9f-6cbb8f462f6b" containerID="13eec92495be8ebec49db0dcb7cefd6b775c9dbdb5a0119ef6e41571ab9eb627" exitCode=0 Nov 03 21:46:56.113006 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:56.112874 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w" event={"ID":"d8f75aac-8457-4abc-ad9f-6cbb8f462f6b","Type":"ContainerDied","Data":"13eec92495be8ebec49db0dcb7cefd6b775c9dbdb5a0119ef6e41571ab9eb627"} Nov 03 21:46:56.288926 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:56.288900 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w" Nov 03 21:46:56.417216 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:56.417181 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8f75aac-8457-4abc-ad9f-6cbb8f462f6b-kube-api-access-26zsn\" (UniqueName: \"kubernetes.io/projected/d8f75aac-8457-4abc-ad9f-6cbb8f462f6b-kube-api-access-26zsn\") pod \"d8f75aac-8457-4abc-ad9f-6cbb8f462f6b\" (UID: \"d8f75aac-8457-4abc-ad9f-6cbb8f462f6b\") " Nov 03 21:46:56.417402 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:56.417366 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8f75aac-8457-4abc-ad9f-6cbb8f462f6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8f75aac-8457-4abc-ad9f-6cbb8f462f6b-plugins-config-volume\") pod \"d8f75aac-8457-4abc-ad9f-6cbb8f462f6b\" (UID: \"d8f75aac-8457-4abc-ad9f-6cbb8f462f6b\") " Nov 03 21:46:56.417805 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:56.417771 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8f75aac-8457-4abc-ad9f-6cbb8f462f6b-plugins-config-volume" pod "d8f75aac-8457-4abc-ad9f-6cbb8f462f6b" (UID: "d8f75aac-8457-4abc-ad9f-6cbb8f462f6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:46:56.417943 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:56.417894 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8f75aac-8457-4abc-ad9f-6cbb8f462f6b-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:46:56.419668 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:56.419634 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f75aac-8457-4abc-ad9f-6cbb8f462f6b-kube-api-access-26zsn" pod "d8f75aac-8457-4abc-ad9f-6cbb8f462f6b" (UID: "d8f75aac-8457-4abc-ad9f-6cbb8f462f6b"). InnerVolumeSpecName "kube-api-access-26zsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:46:56.519153 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:56.519035 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26zsn\" (UniqueName: \"kubernetes.io/projected/d8f75aac-8457-4abc-ad9f-6cbb8f462f6b-kube-api-access-26zsn\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:46:57.149099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:57.146508 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w" podUID="d8f75aac-8457-4abc-ad9f-6cbb8f462f6b" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.168:50052\" within 1s: context deadline exceeded" Nov 03 21:46:57.149099 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:46:57.146936 5615 logging.go:55] [core] [Channel #6182 SubChannel #6183]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.168:50052", ServerName: "10.134.0.168:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.168:50052: operation was canceled" Nov 03 21:46:57.190375 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:57.190324 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w" event={"ID":"d8f75aac-8457-4abc-ad9f-6cbb8f462f6b","Type":"ContainerDied","Data":"577e7f9b09dd14df816034ff5d48e538fd19007151ef20a376324ae3e02a28a1"} Nov 03 21:46:57.190613 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:57.190410 5615 scope.go:122] "RemoveContainer" containerID="13eec92495be8ebec49db0dcb7cefd6b775c9dbdb5a0119ef6e41571ab9eb627" Nov 03 21:46:57.190613 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:57.190582 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w" Nov 03 21:46:57.371307 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:57.370057 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w"] Nov 03 21:46:57.433154 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:57.432533 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a3dc77-7bc9d6786c-99b2w"] Nov 03 21:46:57.653023 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:46:57.652988 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8f75aac-8457-4abc-ad9f-6cbb8f462f6b" path="/var/lib/kubelet/pods/d8f75aac-8457-4abc-ad9f-6cbb8f462f6b/volumes" Nov 03 21:47:07.863379 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:07.863275 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6"] Nov 03 21:47:07.863897 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:07.863596 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" podUID="cc56f893-acbf-4db9-9a61-b7c4c0df44a7" containerName="istio-proxy" containerID="cri-o://0bbff94ce9766fc230fc8ea0b02250f0605f789c8be263568aa3ec7603801b53" gracePeriod=30 Nov 03 21:47:07.870771 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:07.870735 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt"] Nov 03 21:47:07.871038 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:07.870974 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" podUID="faa117f5-40a0-439f-8008-2264c4c8573c" containerName="istio-proxy" containerID="cri-o://7807dd9619c813a6a82cfb06efe0e6d32941e8996110e1b27c848b7e5925216e" gracePeriod=30 Nov 03 21:47:10.646605 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:47:10.646568 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:47:12.231180 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:12.231040 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8"] Nov 03 21:47:12.231713 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:12.231290 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" podUID="5b1a343a-cae6-4551-b189-bb6b0e978ba4" containerName="istio-proxy" containerID="cri-o://993f796158aadb4dc3d38ff297d56f07862f970abfe91782720ee107fa388f84" gracePeriod=30 Nov 03 21:47:12.825639 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:12.825600 5615 patch_prober.go:29] interesting pod/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:47:12.825841 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:12.825705 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" podUID="5b1a343a-cae6-4551-b189-bb6b0e978ba4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:47:13.265160 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.264771 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:47:13.268030 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.267988 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:47:13.377952 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.377860 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istio-podinfo\") pod \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " Nov 03 21:47:13.377952 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.377910 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faa117f5-40a0-439f-8008-2264c4c8573c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa117f5-40a0-439f-8008-2264c4c8573c-istiod-ca-cert\") pod \"faa117f5-40a0-439f-8008-2264c4c8573c\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " Nov 03 21:47:13.378206 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.377982 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istio-token\" (UniqueName: \"kubernetes.io/projected/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istio-token\") pod \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " Nov 03 21:47:13.378206 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.378030 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istiod-ca-cert\") pod \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " Nov 03 21:47:13.378206 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.378053 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-workload-certs\") pod \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " Nov 03 21:47:13.378206 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.378143 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa117f5-40a0-439f-8008-2264c4c8573c-kube-api-access-nl82b\" (UniqueName: \"kubernetes.io/projected/faa117f5-40a0-439f-8008-2264c4c8573c-kube-api-access-nl82b\") pod \"faa117f5-40a0-439f-8008-2264c4c8573c\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " Nov 03 21:47:13.378206 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.378180 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-istio-data\") pod \"faa117f5-40a0-439f-8008-2264c4c8573c\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " Nov 03 21:47:13.378411 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.378224 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-istio-envoy\") pod \"faa117f5-40a0-439f-8008-2264c4c8573c\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " Nov 03 21:47:13.378411 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.378242 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-credential-socket\") pod \"faa117f5-40a0-439f-8008-2264c4c8573c\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " Nov 03 21:47:13.378411 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.378287 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-kube-api-access-cfvnp\" (UniqueName: \"kubernetes.io/projected/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-kube-api-access-cfvnp\") pod \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " Nov 03 21:47:13.378411 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.378285 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa117f5-40a0-439f-8008-2264c4c8573c-istiod-ca-cert" pod "faa117f5-40a0-439f-8008-2264c4c8573c" (UID: "faa117f5-40a0-439f-8008-2264c4c8573c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:47:13.378411 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.378313 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-workload-certs\") pod \"faa117f5-40a0-439f-8008-2264c4c8573c\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " Nov 03 21:47:13.378411 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.378366 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istio-data\") pod \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " Nov 03 21:47:13.378603 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.378432 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-workload-socket\") pod \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " Nov 03 21:47:13.378603 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.378460 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/faa117f5-40a0-439f-8008-2264c4c8573c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa117f5-40a0-439f-8008-2264c4c8573c-istio-podinfo\") pod \"faa117f5-40a0-439f-8008-2264c4c8573c\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " Nov 03 21:47:13.378603 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.378489 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-istio-data" pod "faa117f5-40a0-439f-8008-2264c4c8573c" (UID: "faa117f5-40a0-439f-8008-2264c4c8573c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:13.378603 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.378538 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-workload-socket\") pod \"faa117f5-40a0-439f-8008-2264c4c8573c\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " Nov 03 21:47:13.378736 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.378623 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-credential-socket\") pod \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " Nov 03 21:47:13.378736 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.378684 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa117f5-40a0-439f-8008-2264c4c8573c-istio-token\" (UniqueName: \"kubernetes.io/projected/faa117f5-40a0-439f-8008-2264c4c8573c-istio-token\") pod \"faa117f5-40a0-439f-8008-2264c4c8573c\" (UID: \"faa117f5-40a0-439f-8008-2264c4c8573c\") " Nov 03 21:47:13.379261 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.378858 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-credential-socket" pod "faa117f5-40a0-439f-8008-2264c4c8573c" (UID: "faa117f5-40a0-439f-8008-2264c4c8573c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:13.379261 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.378866 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istio-envoy\") pod \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\" (UID: \"cc56f893-acbf-4db9-9a61-b7c4c0df44a7\") " Nov 03 21:47:13.379261 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.378932 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-workload-socket" pod "faa117f5-40a0-439f-8008-2264c4c8573c" (UID: "faa117f5-40a0-439f-8008-2264c4c8573c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:13.379261 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.378984 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-workload-certs" pod "faa117f5-40a0-439f-8008-2264c4c8573c" (UID: "faa117f5-40a0-439f-8008-2264c4c8573c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:13.379261 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.379136 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-workload-certs" pod "cc56f893-acbf-4db9-9a61-b7c4c0df44a7" (UID: "cc56f893-acbf-4db9-9a61-b7c4c0df44a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:13.379261 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.379231 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istio-data" pod "cc56f893-acbf-4db9-9a61-b7c4c0df44a7" (UID: "cc56f893-acbf-4db9-9a61-b7c4c0df44a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:13.379261 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.379253 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-credential-socket" pod "cc56f893-acbf-4db9-9a61-b7c4c0df44a7" (UID: "cc56f893-acbf-4db9-9a61-b7c4c0df44a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:13.379728 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.379641 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:13.379728 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.379665 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:13.379728 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.379718 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:13.379910 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.379735 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:13.379910 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.379749 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:13.379910 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.379765 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:13.379910 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.379806 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa117f5-40a0-439f-8008-2264c4c8573c-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:13.379910 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.379821 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:13.380346 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.380215 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istiod-ca-cert" pod "cc56f893-acbf-4db9-9a61-b7c4c0df44a7" (UID: "cc56f893-acbf-4db9-9a61-b7c4c0df44a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:47:13.380346 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.380304 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-workload-socket" pod "cc56f893-acbf-4db9-9a61-b7c4c0df44a7" (UID: "cc56f893-acbf-4db9-9a61-b7c4c0df44a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:13.381969 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.381927 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa117f5-40a0-439f-8008-2264c4c8573c-istio-token" pod "faa117f5-40a0-439f-8008-2264c4c8573c" (UID: "faa117f5-40a0-439f-8008-2264c4c8573c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:13.382393 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.382360 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istio-token" pod "cc56f893-acbf-4db9-9a61-b7c4c0df44a7" (UID: "cc56f893-acbf-4db9-9a61-b7c4c0df44a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:13.382690 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.382642 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa117f5-40a0-439f-8008-2264c4c8573c-kube-api-access-nl82b" pod "faa117f5-40a0-439f-8008-2264c4c8573c" (UID: "faa117f5-40a0-439f-8008-2264c4c8573c"). InnerVolumeSpecName "kube-api-access-nl82b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:13.382785 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.382764 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/faa117f5-40a0-439f-8008-2264c4c8573c-istio-podinfo" pod "faa117f5-40a0-439f-8008-2264c4c8573c" (UID: "faa117f5-40a0-439f-8008-2264c4c8573c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:47:13.383277 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.383241 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-istio-envoy" pod "faa117f5-40a0-439f-8008-2264c4c8573c" (UID: "faa117f5-40a0-439f-8008-2264c4c8573c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:13.383485 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.383459 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istio-envoy" pod "cc56f893-acbf-4db9-9a61-b7c4c0df44a7" (UID: "cc56f893-acbf-4db9-9a61-b7c4c0df44a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:13.383558 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.383469 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-kube-api-access-cfvnp" pod "cc56f893-acbf-4db9-9a61-b7c4c0df44a7" (UID: "cc56f893-acbf-4db9-9a61-b7c4c0df44a7"). InnerVolumeSpecName "kube-api-access-cfvnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:13.383686 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.383672 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istio-podinfo" pod "cc56f893-acbf-4db9-9a61-b7c4c0df44a7" (UID: "cc56f893-acbf-4db9-9a61-b7c4c0df44a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:47:13.436618 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.436578 5615 generic.go:356] "Generic (PLEG): container finished" podID="cc56f893-acbf-4db9-9a61-b7c4c0df44a7" containerID="0bbff94ce9766fc230fc8ea0b02250f0605f789c8be263568aa3ec7603801b53" exitCode=0 Nov 03 21:47:13.436782 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.436661 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" event={"ID":"cc56f893-acbf-4db9-9a61-b7c4c0df44a7","Type":"ContainerDied","Data":"0bbff94ce9766fc230fc8ea0b02250f0605f789c8be263568aa3ec7603801b53"} Nov 03 21:47:13.436782 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.436765 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" event={"ID":"cc56f893-acbf-4db9-9a61-b7c4c0df44a7","Type":"ContainerDied","Data":"1fac1bfefa3875a58a3fe68e0791571bb6bacac777ae45d2d5f786d687cd6af8"} Nov 03 21:47:13.436858 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.436800 5615 scope.go:122] "RemoveContainer" containerID="0bbff94ce9766fc230fc8ea0b02250f0605f789c8be263568aa3ec7603801b53" Nov 03 21:47:13.437452 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.437426 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6" Nov 03 21:47:13.439133 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.439106 5615 generic.go:356] "Generic (PLEG): container finished" podID="faa117f5-40a0-439f-8008-2264c4c8573c" containerID="7807dd9619c813a6a82cfb06efe0e6d32941e8996110e1b27c848b7e5925216e" exitCode=0 Nov 03 21:47:13.439292 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.439275 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" event={"ID":"faa117f5-40a0-439f-8008-2264c4c8573c","Type":"ContainerDied","Data":"7807dd9619c813a6a82cfb06efe0e6d32941e8996110e1b27c848b7e5925216e"} Nov 03 21:47:13.439363 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.439322 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" event={"ID":"faa117f5-40a0-439f-8008-2264c4c8573c","Type":"ContainerDied","Data":"b9828a243b51b93e9b5aad68e2f969f9e636f90510417137e8d5bd2b8f271dbd"} Nov 03 21:47:13.439422 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.439395 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt" Nov 03 21:47:13.456464 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.455337 5615 scope.go:122] "RemoveContainer" containerID="0bbff94ce9766fc230fc8ea0b02250f0605f789c8be263568aa3ec7603801b53" Nov 03 21:47:13.456464 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:47:13.455790 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bbff94ce9766fc230fc8ea0b02250f0605f789c8be263568aa3ec7603801b53\": container with ID starting with 0bbff94ce9766fc230fc8ea0b02250f0605f789c8be263568aa3ec7603801b53 not found: ID does not exist" containerID="0bbff94ce9766fc230fc8ea0b02250f0605f789c8be263568aa3ec7603801b53" Nov 03 21:47:13.456464 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.455848 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bbff94ce9766fc230fc8ea0b02250f0605f789c8be263568aa3ec7603801b53"} err="failed to get container status \"0bbff94ce9766fc230fc8ea0b02250f0605f789c8be263568aa3ec7603801b53\": rpc error: code = NotFound desc = could not find container \"0bbff94ce9766fc230fc8ea0b02250f0605f789c8be263568aa3ec7603801b53\": container with ID starting with 0bbff94ce9766fc230fc8ea0b02250f0605f789c8be263568aa3ec7603801b53 not found: ID does not exist" Nov 03 21:47:13.456464 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.455878 5615 scope.go:122] "RemoveContainer" containerID="7807dd9619c813a6a82cfb06efe0e6d32941e8996110e1b27c848b7e5925216e" Nov 03 21:47:13.473182 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.473157 5615 scope.go:122] "RemoveContainer" containerID="7807dd9619c813a6a82cfb06efe0e6d32941e8996110e1b27c848b7e5925216e" Nov 03 21:47:13.473542 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:47:13.473504 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7807dd9619c813a6a82cfb06efe0e6d32941e8996110e1b27c848b7e5925216e\": container with ID starting with 7807dd9619c813a6a82cfb06efe0e6d32941e8996110e1b27c848b7e5925216e not found: ID does not exist" containerID="7807dd9619c813a6a82cfb06efe0e6d32941e8996110e1b27c848b7e5925216e" Nov 03 21:47:13.473542 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.473533 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7807dd9619c813a6a82cfb06efe0e6d32941e8996110e1b27c848b7e5925216e"} err="failed to get container status \"7807dd9619c813a6a82cfb06efe0e6d32941e8996110e1b27c848b7e5925216e\": rpc error: code = NotFound desc = could not find container \"7807dd9619c813a6a82cfb06efe0e6d32941e8996110e1b27c848b7e5925216e\": container with ID starting with 7807dd9619c813a6a82cfb06efe0e6d32941e8996110e1b27c848b7e5925216e not found: ID does not exist" Nov 03 21:47:13.481487 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.481465 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfvnp\" (UniqueName: \"kubernetes.io/projected/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-kube-api-access-cfvnp\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:13.481620 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.481488 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:13.481620 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.481501 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa117f5-40a0-439f-8008-2264c4c8573c-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:13.481620 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.481510 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa117f5-40a0-439f-8008-2264c4c8573c-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:13.481620 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.481518 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:13.481620 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.481527 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:13.481620 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.481535 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:13.481620 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.481543 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc56f893-acbf-4db9-9a61-b7c4c0df44a7-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:13.481620 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.481560 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl82b\" (UniqueName: \"kubernetes.io/projected/faa117f5-40a0-439f-8008-2264c4c8573c-kube-api-access-nl82b\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:13.481620 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.481568 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa117f5-40a0-439f-8008-2264c4c8573c-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:13.641624 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.641524 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt"] Nov 03 21:47:13.652753 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.652721 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-z8sjt"] Nov 03 21:47:13.749638 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.749603 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6"] Nov 03 21:47:13.756180 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:13.756148 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-qbxt6"] Nov 03 21:47:15.653592 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:15.653542 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc56f893-acbf-4db9-9a61-b7c4c0df44a7" path="/var/lib/kubelet/pods/cc56f893-acbf-4db9-9a61-b7c4c0df44a7/volumes" Nov 03 21:47:15.654128 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:15.654112 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faa117f5-40a0-439f-8008-2264c4c8573c" path="/var/lib/kubelet/pods/faa117f5-40a0-439f-8008-2264c4c8573c/volumes" Nov 03 21:47:17.472921 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.472886 5615 generic.go:356] "Generic (PLEG): container finished" podID="5b1a343a-cae6-4551-b189-bb6b0e978ba4" containerID="993f796158aadb4dc3d38ff297d56f07862f970abfe91782720ee107fa388f84" exitCode=0 Nov 03 21:47:17.473423 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.472959 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" event={"ID":"5b1a343a-cae6-4551-b189-bb6b0e978ba4","Type":"ContainerDied","Data":"993f796158aadb4dc3d38ff297d56f07862f970abfe91782720ee107fa388f84"} Nov 03 21:47:17.473423 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.473043 5615 scope.go:122] "RemoveContainer" containerID="fc0200cb8c2785af885a6f131d47cfe4d0892b1f1850a60b68dcfcaf6e05cad5" Nov 03 21:47:17.510246 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:47:17.510218 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc0200cb8c2785af885a6f131d47cfe4d0892b1f1850a60b68dcfcaf6e05cad5\": container with ID starting with fc0200cb8c2785af885a6f131d47cfe4d0892b1f1850a60b68dcfcaf6e05cad5 not found: ID does not exist" containerID="fc0200cb8c2785af885a6f131d47cfe4d0892b1f1850a60b68dcfcaf6e05cad5" Nov 03 21:47:17.521964 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.521935 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:47:17.664782 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.664756 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istio-data\") pod \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " Nov 03 21:47:17.664997 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.664804 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istio-token\" (UniqueName: \"kubernetes.io/projected/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istio-token\") pod \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " Nov 03 21:47:17.664997 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.664904 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-workload-certs\") pod \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " Nov 03 21:47:17.664997 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.664938 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istio-envoy\") pod \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " Nov 03 21:47:17.664997 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.664968 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istiod-ca-cert\") pod \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " Nov 03 21:47:17.664997 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.664990 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istio-podinfo\") pod \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " Nov 03 21:47:17.665327 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.665007 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b1a343a-cae6-4551-b189-bb6b0e978ba4-kube-api-access-7ssg8\" (UniqueName: \"kubernetes.io/projected/5b1a343a-cae6-4551-b189-bb6b0e978ba4-kube-api-access-7ssg8\") pod \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " Nov 03 21:47:17.665327 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.665036 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istio-data" pod "5b1a343a-cae6-4551-b189-bb6b0e978ba4" (UID: "5b1a343a-cae6-4551-b189-bb6b0e978ba4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:17.665327 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.665125 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-credential-socket\") pod \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " Nov 03 21:47:17.665327 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.665210 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-workload-socket\") pod \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\" (UID: \"5b1a343a-cae6-4551-b189-bb6b0e978ba4\") " Nov 03 21:47:17.665327 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.665297 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-workload-certs" pod "5b1a343a-cae6-4551-b189-bb6b0e978ba4" (UID: "5b1a343a-cae6-4551-b189-bb6b0e978ba4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:17.665596 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.665517 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-credential-socket" pod "5b1a343a-cae6-4551-b189-bb6b0e978ba4" (UID: "5b1a343a-cae6-4551-b189-bb6b0e978ba4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:17.665596 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.665541 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istiod-ca-cert" pod "5b1a343a-cae6-4551-b189-bb6b0e978ba4" (UID: "5b1a343a-cae6-4551-b189-bb6b0e978ba4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:47:17.665775 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.665732 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-workload-socket" pod "5b1a343a-cae6-4551-b189-bb6b0e978ba4" (UID: "5b1a343a-cae6-4551-b189-bb6b0e978ba4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:17.665931 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.665911 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:17.665975 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.665937 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:17.665975 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.665955 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:17.665975 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.665968 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:17.666099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.665985 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:17.667307 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.667285 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istio-podinfo" pod "5b1a343a-cae6-4551-b189-bb6b0e978ba4" (UID: "5b1a343a-cae6-4551-b189-bb6b0e978ba4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:47:17.667407 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.667335 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b1a343a-cae6-4551-b189-bb6b0e978ba4-kube-api-access-7ssg8" pod "5b1a343a-cae6-4551-b189-bb6b0e978ba4" (UID: "5b1a343a-cae6-4551-b189-bb6b0e978ba4"). InnerVolumeSpecName "kube-api-access-7ssg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:17.667478 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.667449 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istio-envoy" pod "5b1a343a-cae6-4551-b189-bb6b0e978ba4" (UID: "5b1a343a-cae6-4551-b189-bb6b0e978ba4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:17.668051 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.668034 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istio-token" pod "5b1a343a-cae6-4551-b189-bb6b0e978ba4" (UID: "5b1a343a-cae6-4551-b189-bb6b0e978ba4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:17.767354 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.767269 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:17.767354 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.767303 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:17.767354 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.767313 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b1a343a-cae6-4551-b189-bb6b0e978ba4-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:17.767354 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:17.767322 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ssg8\" (UniqueName: \"kubernetes.io/projected/5b1a343a-cae6-4551-b189-bb6b0e978ba4-kube-api-access-7ssg8\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:18.481599 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:18.481563 5615 generic.go:356] "Generic (PLEG): container finished" podID="4615ffcf-3653-436d-bfc4-ecda29430be4" containerID="afa3e57ac5ee5015b83f3fa5cfa7dd3243592976bed789f6ae5eebc2c7787bf1" exitCode=0 Nov 03 21:47:18.482097 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:18.481626 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" event={"ID":"4615ffcf-3653-436d-bfc4-ecda29430be4","Type":"ContainerDied","Data":"afa3e57ac5ee5015b83f3fa5cfa7dd3243592976bed789f6ae5eebc2c7787bf1"} Nov 03 21:47:18.482224 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:18.482198 5615 scope.go:122] "RemoveContainer" containerID="afa3e57ac5ee5015b83f3fa5cfa7dd3243592976bed789f6ae5eebc2c7787bf1" Nov 03 21:47:18.483188 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:18.483164 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" event={"ID":"5b1a343a-cae6-4551-b189-bb6b0e978ba4","Type":"ContainerDied","Data":"8fd6bc341b0a84d7011fc651fac3e54544d269f9e728de70c85681c3c7879d66"} Nov 03 21:47:18.483273 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:18.483186 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8" Nov 03 21:47:18.483273 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:18.483220 5615 scope.go:122] "RemoveContainer" containerID="993f796158aadb4dc3d38ff297d56f07862f970abfe91782720ee107fa388f84" Nov 03 21:47:18.484172 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:18.484134 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:47:18.484289 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:18.484231 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:47:18.484289 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:18.484278 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:47:18.760916 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:18.760881 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8"] Nov 03 21:47:18.767811 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:18.767780 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d5e2db-openshift-default-6c87cf7865-8kvd8"] Nov 03 21:47:19.492000 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:19.491960 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" event={"ID":"4615ffcf-3653-436d-bfc4-ecda29430be4","Type":"ContainerStarted","Data":"f00c6fc7d412af44722d22ce3c6da10f1ae3263774686edffd74405cf640a33d"} Nov 03 21:47:19.652932 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:19.652892 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b1a343a-cae6-4551-b189-bb6b0e978ba4" path="/var/lib/kubelet/pods/5b1a343a-cae6-4551-b189-bb6b0e978ba4/volumes" Nov 03 21:47:20.236714 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:20.236684 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:47:20.236892 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:20.236859 5615 patch_prober.go:29] interesting pod/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 03 21:47:20.236948 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:20.236902 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" podUID="4615ffcf-3653-436d-bfc4-ecda29430be4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 03 21:47:21.236843 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:21.236806 5615 patch_prober.go:29] interesting pod/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 03 21:47:21.237322 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:21.236870 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" podUID="4615ffcf-3653-436d-bfc4-ecda29430be4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 03 21:47:22.246100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:22.242174 5615 patch_prober.go:29] interesting pod/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 03 21:47:22.246100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:22.242233 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" podUID="4615ffcf-3653-436d-bfc4-ecda29430be4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 03 21:47:23.236354 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:23.236321 5615 patch_prober.go:29] interesting pod/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 03 21:47:23.236564 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:23.236390 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" podUID="4615ffcf-3653-436d-bfc4-ecda29430be4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 03 21:47:24.236697 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:24.236658 5615 patch_prober.go:29] interesting pod/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 03 21:47:24.237166 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:24.236728 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" podUID="4615ffcf-3653-436d-bfc4-ecda29430be4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 03 21:47:25.281437 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:25.281393 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:47:25.282608 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:25.282581 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:47:25.285454 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:47:25.285433 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:47:40.619789 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:47:40.619743 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:48:01.425422 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:01.425374 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9"] Nov 03 21:48:01.426057 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:01.425646 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9" podUID="79d06746-463e-4043-872b-6f64949dccc8" containerName="maas-api" containerID="cri-o://66083bdfca353ea00e803cfd542ab34c1bfcd6d79328380e8e59c93aa344d898" gracePeriod=30 Nov 03 21:48:01.524849 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:01.524808 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9" podUID="79d06746-463e-4043-872b-6f64949dccc8" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.169:8443/health\": dial tcp 10.134.0.169:8443: connect: connection refused" Nov 03 21:48:01.738291 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:01.738261 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9" Nov 03 21:48:01.825110 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:01.822271 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79d06746-463e-4043-872b-6f64949dccc8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79d06746-463e-4043-872b-6f64949dccc8-openshift-service-ca\") pod \"79d06746-463e-4043-872b-6f64949dccc8\" (UID: \"79d06746-463e-4043-872b-6f64949dccc8\") " Nov 03 21:48:01.825110 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:01.822322 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79d06746-463e-4043-872b-6f64949dccc8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79d06746-463e-4043-872b-6f64949dccc8-maas-api-tls\") pod \"79d06746-463e-4043-872b-6f64949dccc8\" (UID: \"79d06746-463e-4043-872b-6f64949dccc8\") " Nov 03 21:48:01.825110 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:01.822490 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d06746-463e-4043-872b-6f64949dccc8-kube-api-access-2w7cw\" (UniqueName: \"kubernetes.io/projected/79d06746-463e-4043-872b-6f64949dccc8-kube-api-access-2w7cw\") pod \"79d06746-463e-4043-872b-6f64949dccc8\" (UID: \"79d06746-463e-4043-872b-6f64949dccc8\") " Nov 03 21:48:01.825110 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:01.822535 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79d06746-463e-4043-872b-6f64949dccc8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/79d06746-463e-4043-872b-6f64949dccc8-metrics-certs\") pod \"79d06746-463e-4043-872b-6f64949dccc8\" (UID: \"79d06746-463e-4043-872b-6f64949dccc8\") " Nov 03 21:48:01.834140 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:01.834044 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d06746-463e-4043-872b-6f64949dccc8-openshift-service-ca" pod "79d06746-463e-4043-872b-6f64949dccc8" (UID: "79d06746-463e-4043-872b-6f64949dccc8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:01.834962 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:01.834928 5615 generic.go:356] "Generic (PLEG): container finished" podID="79d06746-463e-4043-872b-6f64949dccc8" containerID="66083bdfca353ea00e803cfd542ab34c1bfcd6d79328380e8e59c93aa344d898" exitCode=0 Nov 03 21:48:01.835424 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:01.835397 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9" event={"ID":"79d06746-463e-4043-872b-6f64949dccc8","Type":"ContainerDied","Data":"66083bdfca353ea00e803cfd542ab34c1bfcd6d79328380e8e59c93aa344d898"} Nov 03 21:48:01.835815 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:01.835790 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9" event={"ID":"79d06746-463e-4043-872b-6f64949dccc8","Type":"ContainerDied","Data":"34c56e5d48d0637287f3167c60889993cbd3486bba00cf3b57fe620ef51e4315"} Nov 03 21:48:01.835943 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:01.835928 5615 scope.go:122] "RemoveContainer" containerID="66083bdfca353ea00e803cfd542ab34c1bfcd6d79328380e8e59c93aa344d898" Nov 03 21:48:01.836384 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:01.836365 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9" Nov 03 21:48:01.854331 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:01.854258 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79d06746-463e-4043-872b-6f64949dccc8-maas-api-tls" pod "79d06746-463e-4043-872b-6f64949dccc8" (UID: "79d06746-463e-4043-872b-6f64949dccc8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:48:01.858715 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:01.858604 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d06746-463e-4043-872b-6f64949dccc8-kube-api-access-2w7cw" pod "79d06746-463e-4043-872b-6f64949dccc8" (UID: "79d06746-463e-4043-872b-6f64949dccc8"). InnerVolumeSpecName "kube-api-access-2w7cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:01.865131 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:01.859180 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79d06746-463e-4043-872b-6f64949dccc8-metrics-certs" pod "79d06746-463e-4043-872b-6f64949dccc8" (UID: "79d06746-463e-4043-872b-6f64949dccc8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:48:01.879765 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:01.879505 5615 scope.go:122] "RemoveContainer" containerID="66083bdfca353ea00e803cfd542ab34c1bfcd6d79328380e8e59c93aa344d898" Nov 03 21:48:01.880195 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:48:01.880164 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66083bdfca353ea00e803cfd542ab34c1bfcd6d79328380e8e59c93aa344d898\": container with ID starting with 66083bdfca353ea00e803cfd542ab34c1bfcd6d79328380e8e59c93aa344d898 not found: ID does not exist" containerID="66083bdfca353ea00e803cfd542ab34c1bfcd6d79328380e8e59c93aa344d898" Nov 03 21:48:01.880300 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:01.880207 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66083bdfca353ea00e803cfd542ab34c1bfcd6d79328380e8e59c93aa344d898"} err="failed to get container status \"66083bdfca353ea00e803cfd542ab34c1bfcd6d79328380e8e59c93aa344d898\": rpc error: code = NotFound desc = could not find container \"66083bdfca353ea00e803cfd542ab34c1bfcd6d79328380e8e59c93aa344d898\": container with ID starting with 66083bdfca353ea00e803cfd542ab34c1bfcd6d79328380e8e59c93aa344d898 not found: ID does not exist" Nov 03 21:48:01.924495 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:01.924454 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79d06746-463e-4043-872b-6f64949dccc8-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:48:01.924495 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:01.924498 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79d06746-463e-4043-872b-6f64949dccc8-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:48:01.924714 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:01.924514 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w7cw\" (UniqueName: \"kubernetes.io/projected/79d06746-463e-4043-872b-6f64949dccc8-kube-api-access-2w7cw\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:48:01.924714 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:01.924529 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79d06746-463e-4043-872b-6f64949dccc8-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:48:02.252069 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:02.252029 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9"] Nov 03 21:48:02.360098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:02.356163 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a3dc77-58b57c87c-gkcm9"] Nov 03 21:48:03.653020 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:03.652984 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79d06746-463e-4043-872b-6f64949dccc8" path="/var/lib/kubelet/pods/79d06746-463e-4043-872b-6f64949dccc8/volumes" Nov 03 21:48:06.564441 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:06.564392 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5"] Nov 03 21:48:06.564923 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:06.564766 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" podUID="93610f99-51b4-40f3-a2b2-19b628d9331f" containerName="istio-proxy" containerID="cri-o://5849084697f08275c04d514e84e7b404822b6ce6a13352d6bcc2402b0bbd6f73" gracePeriod=30 Nov 03 21:48:07.406826 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:07.406785 5615 patch_prober.go:29] interesting pod/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:48:07.407004 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:07.406849 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" podUID="93610f99-51b4-40f3-a2b2-19b628d9331f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:48:10.619654 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:48:10.619611 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:48:11.844509 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.844440 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:48:11.901915 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.901879 5615 generic.go:356] "Generic (PLEG): container finished" podID="93610f99-51b4-40f3-a2b2-19b628d9331f" containerID="5849084697f08275c04d514e84e7b404822b6ce6a13352d6bcc2402b0bbd6f73" exitCode=0 Nov 03 21:48:11.902139 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.901933 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" Nov 03 21:48:11.902139 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.901962 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" event={"ID":"93610f99-51b4-40f3-a2b2-19b628d9331f","Type":"ContainerDied","Data":"5849084697f08275c04d514e84e7b404822b6ce6a13352d6bcc2402b0bbd6f73"} Nov 03 21:48:11.902139 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.902019 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5" event={"ID":"93610f99-51b4-40f3-a2b2-19b628d9331f","Type":"ContainerDied","Data":"9e0c78640c7a092f1d3a9bd311c1e963f614e43b8c62f4377aabf997e57d75fa"} Nov 03 21:48:11.902139 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.902043 5615 scope.go:122] "RemoveContainer" containerID="5849084697f08275c04d514e84e7b404822b6ce6a13352d6bcc2402b0bbd6f73" Nov 03 21:48:11.911973 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.911952 5615 scope.go:122] "RemoveContainer" containerID="5849084697f08275c04d514e84e7b404822b6ce6a13352d6bcc2402b0bbd6f73" Nov 03 21:48:11.912302 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:48:11.912281 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5849084697f08275c04d514e84e7b404822b6ce6a13352d6bcc2402b0bbd6f73\": container with ID starting with 5849084697f08275c04d514e84e7b404822b6ce6a13352d6bcc2402b0bbd6f73 not found: ID does not exist" containerID="5849084697f08275c04d514e84e7b404822b6ce6a13352d6bcc2402b0bbd6f73" Nov 03 21:48:11.912387 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.912315 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5849084697f08275c04d514e84e7b404822b6ce6a13352d6bcc2402b0bbd6f73"} err="failed to get container status \"5849084697f08275c04d514e84e7b404822b6ce6a13352d6bcc2402b0bbd6f73\": rpc error: code = NotFound desc = could not find container \"5849084697f08275c04d514e84e7b404822b6ce6a13352d6bcc2402b0bbd6f73\": container with ID starting with 5849084697f08275c04d514e84e7b404822b6ce6a13352d6bcc2402b0bbd6f73 not found: ID does not exist" Nov 03 21:48:11.966801 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.966763 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93610f99-51b4-40f3-a2b2-19b628d9331f-kube-api-access-gfkwt\" (UniqueName: \"kubernetes.io/projected/93610f99-51b4-40f3-a2b2-19b628d9331f-kube-api-access-gfkwt\") pod \"93610f99-51b4-40f3-a2b2-19b628d9331f\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " Nov 03 21:48:11.967010 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.966823 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-workload-socket\") pod \"93610f99-51b4-40f3-a2b2-19b628d9331f\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " Nov 03 21:48:11.967010 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.966883 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-workload-certs\") pod \"93610f99-51b4-40f3-a2b2-19b628d9331f\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " Nov 03 21:48:11.967010 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.966975 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93610f99-51b4-40f3-a2b2-19b628d9331f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93610f99-51b4-40f3-a2b2-19b628d9331f-istio-podinfo\") pod \"93610f99-51b4-40f3-a2b2-19b628d9331f\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " Nov 03 21:48:11.967010 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.966993 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-istio-data\") pod \"93610f99-51b4-40f3-a2b2-19b628d9331f\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " Nov 03 21:48:11.967277 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.967128 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93610f99-51b4-40f3-a2b2-19b628d9331f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93610f99-51b4-40f3-a2b2-19b628d9331f-istiod-ca-cert\") pod \"93610f99-51b4-40f3-a2b2-19b628d9331f\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " Nov 03 21:48:11.967277 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.967191 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93610f99-51b4-40f3-a2b2-19b628d9331f-istio-token\" (UniqueName: \"kubernetes.io/projected/93610f99-51b4-40f3-a2b2-19b628d9331f-istio-token\") pod \"93610f99-51b4-40f3-a2b2-19b628d9331f\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " Nov 03 21:48:11.967277 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.967260 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-credential-socket\") pod \"93610f99-51b4-40f3-a2b2-19b628d9331f\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " Nov 03 21:48:11.967448 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.967318 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-istio-envoy\") pod \"93610f99-51b4-40f3-a2b2-19b628d9331f\" (UID: \"93610f99-51b4-40f3-a2b2-19b628d9331f\") " Nov 03 21:48:11.967448 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.967254 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-workload-socket" pod "93610f99-51b4-40f3-a2b2-19b628d9331f" (UID: "93610f99-51b4-40f3-a2b2-19b628d9331f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:11.967448 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.967280 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-workload-certs" pod "93610f99-51b4-40f3-a2b2-19b628d9331f" (UID: "93610f99-51b4-40f3-a2b2-19b628d9331f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:11.967448 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.967295 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-istio-data" pod "93610f99-51b4-40f3-a2b2-19b628d9331f" (UID: "93610f99-51b4-40f3-a2b2-19b628d9331f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:11.967639 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.967461 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-credential-socket" pod "93610f99-51b4-40f3-a2b2-19b628d9331f" (UID: "93610f99-51b4-40f3-a2b2-19b628d9331f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:11.967639 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.967513 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93610f99-51b4-40f3-a2b2-19b628d9331f-istiod-ca-cert" pod "93610f99-51b4-40f3-a2b2-19b628d9331f" (UID: "93610f99-51b4-40f3-a2b2-19b628d9331f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:11.967958 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.967926 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:48:11.968126 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.967962 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:48:11.968126 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.967977 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:48:11.968126 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.967992 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93610f99-51b4-40f3-a2b2-19b628d9331f-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:48:11.968126 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.968007 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:48:11.969245 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.969210 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93610f99-51b4-40f3-a2b2-19b628d9331f-kube-api-access-gfkwt" pod "93610f99-51b4-40f3-a2b2-19b628d9331f" (UID: "93610f99-51b4-40f3-a2b2-19b628d9331f"). InnerVolumeSpecName "kube-api-access-gfkwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:11.969503 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.969489 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-istio-envoy" pod "93610f99-51b4-40f3-a2b2-19b628d9331f" (UID: "93610f99-51b4-40f3-a2b2-19b628d9331f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:11.969743 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.969718 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93610f99-51b4-40f3-a2b2-19b628d9331f-istio-token" pod "93610f99-51b4-40f3-a2b2-19b628d9331f" (UID: "93610f99-51b4-40f3-a2b2-19b628d9331f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:11.969743 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:11.969732 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93610f99-51b4-40f3-a2b2-19b628d9331f-istio-podinfo" pod "93610f99-51b4-40f3-a2b2-19b628d9331f" (UID: "93610f99-51b4-40f3-a2b2-19b628d9331f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:48:12.069120 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:12.069060 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfkwt\" (UniqueName: \"kubernetes.io/projected/93610f99-51b4-40f3-a2b2-19b628d9331f-kube-api-access-gfkwt\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:48:12.069120 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:12.069120 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93610f99-51b4-40f3-a2b2-19b628d9331f-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:48:12.069379 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:12.069136 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93610f99-51b4-40f3-a2b2-19b628d9331f-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:48:12.069379 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:12.069153 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93610f99-51b4-40f3-a2b2-19b628d9331f-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:48:12.318260 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:12.318225 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5"] Nov 03 21:48:12.325947 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:12.325913 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-hmtf5"] Nov 03 21:48:13.652546 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:13.652510 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93610f99-51b4-40f3-a2b2-19b628d9331f" path="/var/lib/kubelet/pods/93610f99-51b4-40f3-a2b2-19b628d9331f/volumes" Nov 03 21:48:19.133185 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.133146 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4"] Nov 03 21:48:19.133794 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.133778 5615 cpu_manager.go:414] "containerMap: removing container" podUID="cc56f893-acbf-4db9-9a61-b7c4c0df44a7" containerName="istio-proxy" Nov 03 21:48:19.133862 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.133795 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc56f893-acbf-4db9-9a61-b7c4c0df44a7" containerName="istio-proxy" Nov 03 21:48:19.133862 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.133813 5615 cpu_manager.go:414] "containerMap: removing container" podUID="5b1a343a-cae6-4551-b189-bb6b0e978ba4" containerName="istio-proxy" Nov 03 21:48:19.133862 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.133818 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b1a343a-cae6-4551-b189-bb6b0e978ba4" containerName="istio-proxy" Nov 03 21:48:19.133862 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.133826 5615 cpu_manager.go:414] "containerMap: removing container" podUID="faa117f5-40a0-439f-8008-2264c4c8573c" containerName="istio-proxy" Nov 03 21:48:19.133862 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.133832 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa117f5-40a0-439f-8008-2264c4c8573c" containerName="istio-proxy" Nov 03 21:48:19.133862 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.133841 5615 cpu_manager.go:414] "containerMap: removing container" podUID="d8f75aac-8457-4abc-ad9f-6cbb8f462f6b" containerName="payload-pre-processing" Nov 03 21:48:19.133862 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.133847 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8f75aac-8457-4abc-ad9f-6cbb8f462f6b" containerName="payload-pre-processing" Nov 03 21:48:19.134157 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.133878 5615 cpu_manager.go:414] "containerMap: removing container" podUID="79d06746-463e-4043-872b-6f64949dccc8" containerName="maas-api" Nov 03 21:48:19.134157 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.133886 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d06746-463e-4043-872b-6f64949dccc8" containerName="maas-api" Nov 03 21:48:19.134157 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.133898 5615 cpu_manager.go:414] "containerMap: removing container" podUID="5b1a343a-cae6-4551-b189-bb6b0e978ba4" containerName="istio-proxy" Nov 03 21:48:19.134157 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.133906 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b1a343a-cae6-4551-b189-bb6b0e978ba4" containerName="istio-proxy" Nov 03 21:48:19.134157 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.133913 5615 cpu_manager.go:414] "containerMap: removing container" podUID="93610f99-51b4-40f3-a2b2-19b628d9331f" containerName="istio-proxy" Nov 03 21:48:19.134157 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.133917 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93610f99-51b4-40f3-a2b2-19b628d9331f" containerName="istio-proxy" Nov 03 21:48:19.134157 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.134011 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d06746-463e-4043-872b-6f64949dccc8" containerName="maas-api" Nov 03 21:48:19.134157 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.134020 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b1a343a-cae6-4551-b189-bb6b0e978ba4" containerName="istio-proxy" Nov 03 21:48:19.134157 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.134025 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="93610f99-51b4-40f3-a2b2-19b628d9331f" containerName="istio-proxy" Nov 03 21:48:19.134157 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.134037 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc56f893-acbf-4db9-9a61-b7c4c0df44a7" containerName="istio-proxy" Nov 03 21:48:19.134157 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.134044 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b1a343a-cae6-4551-b189-bb6b0e978ba4" containerName="istio-proxy" Nov 03 21:48:19.134157 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.134049 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa117f5-40a0-439f-8008-2264c4c8573c" containerName="istio-proxy" Nov 03 21:48:19.134157 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.134056 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8f75aac-8457-4abc-ad9f-6cbb8f462f6b" containerName="payload-pre-processing" Nov 03 21:48:19.139559 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.139537 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:19.236571 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.236537 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4cjrg\"" Nov 03 21:48:19.240786 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.240747 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4"] Nov 03 21:48:19.275863 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.275825 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb73420b-7000-4ae1-99de-e72bdd6b847b-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-n5nd4\" (UID: \"cb73420b-7000-4ae1-99de-e72bdd6b847b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:19.276014 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.275869 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9762h\" (UniqueName: \"kubernetes.io/projected/cb73420b-7000-4ae1-99de-e72bdd6b847b-kube-api-access-9762h\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-n5nd4\" (UID: \"cb73420b-7000-4ae1-99de-e72bdd6b847b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:19.276014 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.275971 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb73420b-7000-4ae1-99de-e72bdd6b847b-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-n5nd4\" (UID: \"cb73420b-7000-4ae1-99de-e72bdd6b847b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:19.276140 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.276013 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb73420b-7000-4ae1-99de-e72bdd6b847b-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-n5nd4\" (UID: \"cb73420b-7000-4ae1-99de-e72bdd6b847b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:19.276140 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.276124 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb73420b-7000-4ae1-99de-e72bdd6b847b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-n5nd4\" (UID: \"cb73420b-7000-4ae1-99de-e72bdd6b847b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:19.276237 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.276222 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb73420b-7000-4ae1-99de-e72bdd6b847b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-n5nd4\" (UID: \"cb73420b-7000-4ae1-99de-e72bdd6b847b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:19.276274 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.276261 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb73420b-7000-4ae1-99de-e72bdd6b847b-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-n5nd4\" (UID: \"cb73420b-7000-4ae1-99de-e72bdd6b847b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:19.276394 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.276373 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb73420b-7000-4ae1-99de-e72bdd6b847b-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-n5nd4\" (UID: \"cb73420b-7000-4ae1-99de-e72bdd6b847b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:19.276431 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.276407 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb73420b-7000-4ae1-99de-e72bdd6b847b-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-n5nd4\" (UID: \"cb73420b-7000-4ae1-99de-e72bdd6b847b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:19.378104 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.378032 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb73420b-7000-4ae1-99de-e72bdd6b847b-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-n5nd4\" (UID: \"cb73420b-7000-4ae1-99de-e72bdd6b847b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:19.378331 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.378119 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb73420b-7000-4ae1-99de-e72bdd6b847b-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-n5nd4\" (UID: \"cb73420b-7000-4ae1-99de-e72bdd6b847b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:19.378331 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.378153 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb73420b-7000-4ae1-99de-e72bdd6b847b-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-n5nd4\" (UID: \"cb73420b-7000-4ae1-99de-e72bdd6b847b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:19.378331 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.378190 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9762h\" (UniqueName: \"kubernetes.io/projected/cb73420b-7000-4ae1-99de-e72bdd6b847b-kube-api-access-9762h\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-n5nd4\" (UID: \"cb73420b-7000-4ae1-99de-e72bdd6b847b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:19.378331 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.378256 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb73420b-7000-4ae1-99de-e72bdd6b847b-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-n5nd4\" (UID: \"cb73420b-7000-4ae1-99de-e72bdd6b847b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:19.378331 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.378308 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb73420b-7000-4ae1-99de-e72bdd6b847b-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-n5nd4\" (UID: \"cb73420b-7000-4ae1-99de-e72bdd6b847b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:19.378695 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.378374 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb73420b-7000-4ae1-99de-e72bdd6b847b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-n5nd4\" (UID: \"cb73420b-7000-4ae1-99de-e72bdd6b847b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:19.378695 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.378471 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb73420b-7000-4ae1-99de-e72bdd6b847b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-n5nd4\" (UID: \"cb73420b-7000-4ae1-99de-e72bdd6b847b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:19.378695 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.378488 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb73420b-7000-4ae1-99de-e72bdd6b847b-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-n5nd4\" (UID: \"cb73420b-7000-4ae1-99de-e72bdd6b847b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:19.378695 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.378517 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb73420b-7000-4ae1-99de-e72bdd6b847b-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-n5nd4\" (UID: \"cb73420b-7000-4ae1-99de-e72bdd6b847b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:19.378919 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.378897 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb73420b-7000-4ae1-99de-e72bdd6b847b-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-n5nd4\" (UID: \"cb73420b-7000-4ae1-99de-e72bdd6b847b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:19.379006 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.378975 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb73420b-7000-4ae1-99de-e72bdd6b847b-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-n5nd4\" (UID: \"cb73420b-7000-4ae1-99de-e72bdd6b847b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:19.379006 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.378990 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb73420b-7000-4ae1-99de-e72bdd6b847b-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-n5nd4\" (UID: \"cb73420b-7000-4ae1-99de-e72bdd6b847b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:19.379303 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.379284 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb73420b-7000-4ae1-99de-e72bdd6b847b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-n5nd4\" (UID: \"cb73420b-7000-4ae1-99de-e72bdd6b847b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:19.380967 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.380940 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb73420b-7000-4ae1-99de-e72bdd6b847b-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-n5nd4\" (UID: \"cb73420b-7000-4ae1-99de-e72bdd6b847b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:19.381062 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.381009 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb73420b-7000-4ae1-99de-e72bdd6b847b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-n5nd4\" (UID: \"cb73420b-7000-4ae1-99de-e72bdd6b847b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:19.438433 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.438355 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb73420b-7000-4ae1-99de-e72bdd6b847b-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-n5nd4\" (UID: \"cb73420b-7000-4ae1-99de-e72bdd6b847b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:19.438581 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.438449 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9762h\" (UniqueName: \"kubernetes.io/projected/cb73420b-7000-4ae1-99de-e72bdd6b847b-kube-api-access-9762h\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-n5nd4\" (UID: \"cb73420b-7000-4ae1-99de-e72bdd6b847b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:19.454327 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.454294 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:19.879183 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.879148 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4"] Nov 03 21:48:19.880707 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:48:19.880674 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb73420b_7000_4ae1_99de_e72bdd6b847b.slice/crio-bcf0344986d6cb8f4c4ca53e1223ea22ff213b6650cff4848899ae70085a2dbf WatchSource:0}: Error finding container bcf0344986d6cb8f4c4ca53e1223ea22ff213b6650cff4848899ae70085a2dbf: Status 404 returned error can't find the container with id bcf0344986d6cb8f4c4ca53e1223ea22ff213b6650cff4848899ae70085a2dbf Nov 03 21:48:19.883380 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.883346 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:48:19.883469 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.883415 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:48:19.883469 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.883444 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:48:19.959543 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:19.959507 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" event={"ID":"cb73420b-7000-4ae1-99de-e72bdd6b847b","Type":"ContainerStarted","Data":"bcf0344986d6cb8f4c4ca53e1223ea22ff213b6650cff4848899ae70085a2dbf"} Nov 03 21:48:20.967581 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:20.967537 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" event={"ID":"cb73420b-7000-4ae1-99de-e72bdd6b847b","Type":"ContainerStarted","Data":"c4a30449d05de6e6073dbe47f80fc3beda3e9c93f71850d04b512702b9e516f4"} Nov 03 21:48:21.082501 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:21.082445 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" podStartSLOduration=2.08243049 podStartE2EDuration="2.08243049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:48:21.08129229 +0000 UTC m=+3595.549071016" watchObservedRunningTime="2026-11-03 21:48:21.08243049 +0000 UTC m=+3595.550209212" Nov 03 21:48:21.458514 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:21.458476 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:21.461837 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:21.461482 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 21:48:21.461837 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:21.461562 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" podUID="cb73420b-7000-4ae1-99de-e72bdd6b847b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 21:48:22.455577 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:22.455542 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 21:48:22.455996 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:22.455614 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" podUID="cb73420b-7000-4ae1-99de-e72bdd6b847b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 21:48:23.455479 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:23.455442 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 21:48:23.455701 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:23.455498 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" podUID="cb73420b-7000-4ae1-99de-e72bdd6b847b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 21:48:24.454906 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:24.454867 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 21:48:24.455068 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:24.454937 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" podUID="cb73420b-7000-4ae1-99de-e72bdd6b847b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 21:48:25.454745 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:25.454710 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 21:48:25.455211 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:25.454779 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" podUID="cb73420b-7000-4ae1-99de-e72bdd6b847b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 21:48:26.455489 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:26.455448 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 21:48:26.455912 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:26.455506 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" podUID="cb73420b-7000-4ae1-99de-e72bdd6b847b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 21:48:27.455237 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:27.455203 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 21:48:27.455388 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:27.455267 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" podUID="cb73420b-7000-4ae1-99de-e72bdd6b847b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 21:48:28.455198 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:28.455154 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 21:48:28.455692 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:28.455221 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" podUID="cb73420b-7000-4ae1-99de-e72bdd6b847b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 21:48:29.552527 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:29.552483 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:29.553010 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:29.552987 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:29.553878 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:29.553860 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" Nov 03 21:48:40.664830 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:48:40.664735 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:48:42.474417 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:42.474381 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp"] Nov 03 21:48:42.474848 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:42.474629 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" podUID="5669c7e0-fefa-4212-b3c9-411e65c59fa5" containerName="istio-proxy" containerID="cri-o://8603bb3f51e339bda47af34d6f43ece73cf6fcc0d3063d59faaf1a34f8c664f7" gracePeriod=30 Nov 03 21:48:47.757431 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:47.757405 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:48:47.821539 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:47.821501 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istio-data\") pod \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " Nov 03 21:48:47.821539 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:47.821539 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-credential-socket\") pod \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " Nov 03 21:48:47.821769 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:47.821575 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istio-podinfo\") pod \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " Nov 03 21:48:47.821769 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:47.821625 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istio-token\" (UniqueName: \"kubernetes.io/projected/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istio-token\") pod \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " Nov 03 21:48:47.821769 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:47.821647 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-workload-certs\") pod \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " Nov 03 21:48:47.821769 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:47.821736 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5669c7e0-fefa-4212-b3c9-411e65c59fa5-kube-api-access-cc2bt\" (UniqueName: \"kubernetes.io/projected/5669c7e0-fefa-4212-b3c9-411e65c59fa5-kube-api-access-cc2bt\") pod \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " Nov 03 21:48:47.821769 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:47.821761 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istiod-ca-cert\") pod \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " Nov 03 21:48:47.822036 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:47.821824 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istio-envoy\") pod \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " Nov 03 21:48:47.822036 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:47.821853 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istio-data" pod "5669c7e0-fefa-4212-b3c9-411e65c59fa5" (UID: "5669c7e0-fefa-4212-b3c9-411e65c59fa5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:47.822036 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:47.821870 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-credential-socket" pod "5669c7e0-fefa-4212-b3c9-411e65c59fa5" (UID: "5669c7e0-fefa-4212-b3c9-411e65c59fa5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:47.822036 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:47.821909 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-workload-socket\") pod \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\" (UID: \"5669c7e0-fefa-4212-b3c9-411e65c59fa5\") " Nov 03 21:48:47.822389 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:47.822235 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-workload-certs" pod "5669c7e0-fefa-4212-b3c9-411e65c59fa5" (UID: "5669c7e0-fefa-4212-b3c9-411e65c59fa5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:47.822389 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:47.822253 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istiod-ca-cert" pod "5669c7e0-fefa-4212-b3c9-411e65c59fa5" (UID: "5669c7e0-fefa-4212-b3c9-411e65c59fa5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:47.822568 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:47.822414 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-workload-socket" pod "5669c7e0-fefa-4212-b3c9-411e65c59fa5" (UID: "5669c7e0-fefa-4212-b3c9-411e65c59fa5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:47.822629 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:47.822606 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:48:47.822676 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:47.822629 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:48:47.822676 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:47.822644 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:48:47.822676 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:47.822658 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:48:47.822676 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:47.822672 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:48:47.824157 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:47.824131 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istio-podinfo" pod "5669c7e0-fefa-4212-b3c9-411e65c59fa5" (UID: "5669c7e0-fefa-4212-b3c9-411e65c59fa5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:48:47.824276 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:47.824233 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5669c7e0-fefa-4212-b3c9-411e65c59fa5-kube-api-access-cc2bt" pod "5669c7e0-fefa-4212-b3c9-411e65c59fa5" (UID: "5669c7e0-fefa-4212-b3c9-411e65c59fa5"). InnerVolumeSpecName "kube-api-access-cc2bt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:47.824381 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:47.824363 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istio-token" pod "5669c7e0-fefa-4212-b3c9-411e65c59fa5" (UID: "5669c7e0-fefa-4212-b3c9-411e65c59fa5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:47.824442 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:47.824371 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istio-envoy" pod "5669c7e0-fefa-4212-b3c9-411e65c59fa5" (UID: "5669c7e0-fefa-4212-b3c9-411e65c59fa5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:47.923804 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:47.923710 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc2bt\" (UniqueName: \"kubernetes.io/projected/5669c7e0-fefa-4212-b3c9-411e65c59fa5-kube-api-access-cc2bt\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:48:47.923804 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:47.923746 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:48:47.923804 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:47.923761 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:48:47.923804 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:47.923775 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5669c7e0-fefa-4212-b3c9-411e65c59fa5-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:48:48.177010 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:48.176913 5615 generic.go:356] "Generic (PLEG): container finished" podID="5669c7e0-fefa-4212-b3c9-411e65c59fa5" containerID="8603bb3f51e339bda47af34d6f43ece73cf6fcc0d3063d59faaf1a34f8c664f7" exitCode=0 Nov 03 21:48:48.177010 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:48.176969 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" Nov 03 21:48:48.177280 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:48.176999 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" event={"ID":"5669c7e0-fefa-4212-b3c9-411e65c59fa5","Type":"ContainerDied","Data":"8603bb3f51e339bda47af34d6f43ece73cf6fcc0d3063d59faaf1a34f8c664f7"} Nov 03 21:48:48.177280 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:48.177084 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp" event={"ID":"5669c7e0-fefa-4212-b3c9-411e65c59fa5","Type":"ContainerDied","Data":"d253e56f6659469435634c16addadf68d0434f87ffbacd6d161a05db514ff3b2"} Nov 03 21:48:48.177280 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:48.177124 5615 scope.go:122] "RemoveContainer" containerID="8603bb3f51e339bda47af34d6f43ece73cf6fcc0d3063d59faaf1a34f8c664f7" Nov 03 21:48:48.188474 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:48.188451 5615 scope.go:122] "RemoveContainer" containerID="32109f01797a2eb2b02f8d7e91348bfe43851b371acc790c9183e46ca5118667" Nov 03 21:48:48.198176 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:48.198145 5615 scope.go:122] "RemoveContainer" containerID="8603bb3f51e339bda47af34d6f43ece73cf6fcc0d3063d59faaf1a34f8c664f7" Nov 03 21:48:48.198478 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:48:48.198453 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8603bb3f51e339bda47af34d6f43ece73cf6fcc0d3063d59faaf1a34f8c664f7\": container with ID starting with 8603bb3f51e339bda47af34d6f43ece73cf6fcc0d3063d59faaf1a34f8c664f7 not found: ID does not exist" containerID="8603bb3f51e339bda47af34d6f43ece73cf6fcc0d3063d59faaf1a34f8c664f7" Nov 03 21:48:48.198535 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:48.198491 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8603bb3f51e339bda47af34d6f43ece73cf6fcc0d3063d59faaf1a34f8c664f7"} err="failed to get container status \"8603bb3f51e339bda47af34d6f43ece73cf6fcc0d3063d59faaf1a34f8c664f7\": rpc error: code = NotFound desc = could not find container \"8603bb3f51e339bda47af34d6f43ece73cf6fcc0d3063d59faaf1a34f8c664f7\": container with ID starting with 8603bb3f51e339bda47af34d6f43ece73cf6fcc0d3063d59faaf1a34f8c664f7 not found: ID does not exist" Nov 03 21:48:48.198535 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:48.198523 5615 scope.go:122] "RemoveContainer" containerID="32109f01797a2eb2b02f8d7e91348bfe43851b371acc790c9183e46ca5118667" Nov 03 21:48:48.198812 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:48:48.198791 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32109f01797a2eb2b02f8d7e91348bfe43851b371acc790c9183e46ca5118667\": container with ID starting with 32109f01797a2eb2b02f8d7e91348bfe43851b371acc790c9183e46ca5118667 not found: ID does not exist" containerID="32109f01797a2eb2b02f8d7e91348bfe43851b371acc790c9183e46ca5118667" Nov 03 21:48:48.198914 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:48.198819 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32109f01797a2eb2b02f8d7e91348bfe43851b371acc790c9183e46ca5118667"} err="failed to get container status \"32109f01797a2eb2b02f8d7e91348bfe43851b371acc790c9183e46ca5118667\": rpc error: code = NotFound desc = could not find container \"32109f01797a2eb2b02f8d7e91348bfe43851b371acc790c9183e46ca5118667\": container with ID starting with 32109f01797a2eb2b02f8d7e91348bfe43851b371acc790c9183e46ca5118667 not found: ID does not exist" Nov 03 21:48:48.334186 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:48.334153 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp"] Nov 03 21:48:48.381854 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:48.381816 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-c0fc5f-openshift-default-86f7b8bffc-r2nhp"] Nov 03 21:48:49.651883 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:48:49.651849 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5669c7e0-fefa-4212-b3c9-411e65c59fa5" path="/var/lib/kubelet/pods/5669c7e0-fefa-4212-b3c9-411e65c59fa5/volumes" Nov 03 21:49:10.619799 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:49:10.619753 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:49:34.719813 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.719773 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj"] Nov 03 21:49:34.720499 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.720483 5615 cpu_manager.go:414] "containerMap: removing container" podUID="5669c7e0-fefa-4212-b3c9-411e65c59fa5" containerName="istio-proxy" Nov 03 21:49:34.720554 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.720500 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5669c7e0-fefa-4212-b3c9-411e65c59fa5" containerName="istio-proxy" Nov 03 21:49:34.720554 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.720515 5615 cpu_manager.go:414] "containerMap: removing container" podUID="5669c7e0-fefa-4212-b3c9-411e65c59fa5" containerName="istio-proxy" Nov 03 21:49:34.720554 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.720521 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5669c7e0-fefa-4212-b3c9-411e65c59fa5" containerName="istio-proxy" Nov 03 21:49:34.720660 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.720636 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="5669c7e0-fefa-4212-b3c9-411e65c59fa5" containerName="istio-proxy" Nov 03 21:49:34.720660 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.720644 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="5669c7e0-fefa-4212-b3c9-411e65c59fa5" containerName="istio-proxy" Nov 03 21:49:34.724207 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.724185 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:34.756410 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.756362 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-776jj\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:34.756588 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.756421 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xmjc\" (UniqueName: \"kubernetes.io/projected/7803de3d-9e9c-475a-9227-74535c9c1a68-kube-api-access-6xmjc\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-776jj\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:34.756638 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.756583 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-776jj\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:34.756689 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.756668 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-776jj\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:34.756727 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.756701 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-776jj\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:34.756813 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.756792 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7803de3d-9e9c-475a-9227-74535c9c1a68-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-776jj\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:34.756857 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.756840 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-776jj\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:34.756986 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.756972 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7803de3d-9e9c-475a-9227-74535c9c1a68-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-776jj\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:34.757044 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.757012 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7803de3d-9e9c-475a-9227-74535c9c1a68-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-776jj\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:34.850059 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.850016 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj"] Nov 03 21:49:34.857982 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.857944 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-776jj\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:34.858480 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.858008 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xmjc\" (UniqueName: \"kubernetes.io/projected/7803de3d-9e9c-475a-9227-74535c9c1a68-kube-api-access-6xmjc\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-776jj\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:34.858480 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.858066 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-776jj\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:34.858480 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.858132 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-776jj\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:34.858480 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.858178 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-776jj\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:34.858480 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.858224 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7803de3d-9e9c-475a-9227-74535c9c1a68-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-776jj\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:34.858480 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.858253 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-776jj\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:34.858742 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.858724 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-776jj\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:34.858801 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.858784 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-776jj\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:34.859022 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.858995 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7803de3d-9e9c-475a-9227-74535c9c1a68-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-776jj\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:34.859141 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.859069 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-776jj\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:34.859234 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.859210 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-776jj\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:34.859431 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.859408 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7803de3d-9e9c-475a-9227-74535c9c1a68-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-776jj\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:34.860227 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.860199 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7803de3d-9e9c-475a-9227-74535c9c1a68-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-776jj\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:34.861503 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.861473 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7803de3d-9e9c-475a-9227-74535c9c1a68-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-776jj\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:34.862963 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.862938 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-776jj\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:34.920906 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.920870 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7803de3d-9e9c-475a-9227-74535c9c1a68-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-776jj\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:34.921070 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:34.920998 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xmjc\" (UniqueName: \"kubernetes.io/projected/7803de3d-9e9c-475a-9227-74535c9c1a68-kube-api-access-6xmjc\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-776jj\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:35.038686 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:35.038648 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:35.470112 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:35.470068 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj"] Nov 03 21:49:35.470825 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:35.470789 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:49:35.470941 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:35.470861 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:49:35.470941 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:35.470925 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:49:35.581268 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:35.581213 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" event={"ID":"7803de3d-9e9c-475a-9227-74535c9c1a68","Type":"ContainerStarted","Data":"6329bddbbeea5304c526adfd1a606ee5281f84a849bfa800f9507d1cd87d377d"} Nov 03 21:49:35.581437 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:35.581301 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" event={"ID":"7803de3d-9e9c-475a-9227-74535c9c1a68","Type":"ContainerStarted","Data":"3b84bbc316d370d1da70b100413a3f8cfc99fefc6749c42e472999c0adc58313"} Nov 03 21:49:35.704350 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:35.704288 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" podStartSLOduration=1.70427354 podStartE2EDuration="1.70427354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:35.703094701 +0000 UTC m=+3670.170873425" watchObservedRunningTime="2026-11-03 21:49:35.70427354 +0000 UTC m=+3670.172052316" Nov 03 21:49:36.040296 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:36.040260 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:36.043274 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:36.043225 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-776jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 03 21:49:36.043465 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:36.043309 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" podUID="7803de3d-9e9c-475a-9227-74535c9c1a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 03 21:49:37.039593 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:37.039557 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-776jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 03 21:49:37.039780 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:37.039615 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" podUID="7803de3d-9e9c-475a-9227-74535c9c1a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 03 21:49:38.039760 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:38.039719 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-776jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 03 21:49:38.040199 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:38.039773 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" podUID="7803de3d-9e9c-475a-9227-74535c9c1a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 03 21:49:39.039747 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:39.039710 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-776jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 03 21:49:39.039931 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:39.039781 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" podUID="7803de3d-9e9c-475a-9227-74535c9c1a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 03 21:49:40.038963 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:40.038925 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-776jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 03 21:49:40.039193 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:40.038991 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" podUID="7803de3d-9e9c-475a-9227-74535c9c1a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 03 21:49:40.674555 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:49:40.674519 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:49:40.738724 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:40.738675 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z"] Nov 03 21:49:40.743090 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:40.743055 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:40.767613 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:40.767564 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f2fd230-b1c2-4835-bc30-25437b149c02-istiod-ca-cert\") pod \"e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:40.767776 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:40.767621 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmzzq\" (UniqueName: \"kubernetes.io/projected/0f2fd230-b1c2-4835-bc30-25437b149c02-kube-api-access-zmzzq\") pod \"e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:40.767776 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:40.767748 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-workload-socket\") pod \"e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:40.767850 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:40.767790 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f2fd230-b1c2-4835-bc30-25437b149c02-istio-podinfo\") pod \"e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:40.767901 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:40.767887 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-workload-certs\") pod \"e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:40.768006 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:40.767993 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-credential-socket\") pod \"e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:40.768084 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:40.768023 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-istio-envoy\") pod \"e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:40.768084 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:40.768042 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f2fd230-b1c2-4835-bc30-25437b149c02-istio-token\") pod \"e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:40.768183 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:40.768147 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-istio-data\") pod \"e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:40.805206 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:40.805172 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-965fe8-openshift-default-dockercfg-rckm6\"" Nov 03 21:49:40.872106 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:40.870530 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-istio-data\") pod \"e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:40.872106 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:40.870620 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f2fd230-b1c2-4835-bc30-25437b149c02-istiod-ca-cert\") pod \"e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:40.872106 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:40.870647 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmzzq\" (UniqueName: \"kubernetes.io/projected/0f2fd230-b1c2-4835-bc30-25437b149c02-kube-api-access-zmzzq\") pod \"e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:40.872106 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:40.870681 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-workload-socket\") pod \"e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:40.872106 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:40.870724 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f2fd230-b1c2-4835-bc30-25437b149c02-istio-podinfo\") pod \"e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:40.872106 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:40.870834 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-workload-certs\") pod \"e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:40.872106 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:40.870956 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-credential-socket\") pod \"e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:40.872106 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:40.870995 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-istio-envoy\") pod \"e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:40.872106 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:40.871023 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f2fd230-b1c2-4835-bc30-25437b149c02-istio-token\") pod \"e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:40.872106 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:40.871765 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-istio-data\") pod \"e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:40.872800 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:40.872388 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f2fd230-b1c2-4835-bc30-25437b149c02-istiod-ca-cert\") pod \"e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:40.924101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:40.872910 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-workload-socket\") pod \"e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:40.924101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:40.884728 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-credential-socket\") pod \"e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:40.924101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:40.886228 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-workload-certs\") pod \"e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:40.996887 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:40.993582 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z"] Nov 03 21:49:41.047738 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:41.038691 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmzzq\" (UniqueName: \"kubernetes.io/projected/0f2fd230-b1c2-4835-bc30-25437b149c02-kube-api-access-zmzzq\") pod \"e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:41.047738 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:41.039002 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-776jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 03 21:49:41.047738 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:41.039045 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" podUID="7803de3d-9e9c-475a-9227-74535c9c1a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 03 21:49:41.047738 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:41.039989 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f2fd230-b1c2-4835-bc30-25437b149c02-istio-podinfo\") pod \"e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:41.047738 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:41.040373 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-istio-envoy\") pod \"e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:41.047738 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:41.040656 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f2fd230-b1c2-4835-bc30-25437b149c02-istio-token\") pod \"e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:41.057188 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:41.057151 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:41.523375 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:41.523262 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z"] Nov 03 21:49:41.525051 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:49:41.524809 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f2fd230_b1c2_4835_bc30_25437b149c02.slice/crio-2a37c2f3fb7fd45f549813f3b87b31241010fff21153e5793410c60caf291350 WatchSource:0}: Error finding container 2a37c2f3fb7fd45f549813f3b87b31241010fff21153e5793410c60caf291350: Status 404 returned error can't find the container with id 2a37c2f3fb7fd45f549813f3b87b31241010fff21153e5793410c60caf291350 Nov 03 21:49:41.527874 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:41.527840 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:49:41.527960 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:41.527912 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:49:41.527960 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:41.527941 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:49:41.668361 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:41.668325 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" event={"ID":"0f2fd230-b1c2-4835-bc30-25437b149c02","Type":"ContainerStarted","Data":"858c66df7ca402d81c7079c5df4851c1c0b37b2f543db77f6cc21bdd9eef5cc7"} Nov 03 21:49:41.668639 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:41.668389 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" event={"ID":"0f2fd230-b1c2-4835-bc30-25437b149c02","Type":"ContainerStarted","Data":"2a37c2f3fb7fd45f549813f3b87b31241010fff21153e5793410c60caf291350"} Nov 03 21:49:41.791795 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:41.791677 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" podStartSLOduration=1.79165827 podStartE2EDuration="1.79165827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:41.790223998 +0000 UTC m=+3676.258002720" watchObservedRunningTime="2026-11-03 21:49:41.79165827 +0000 UTC m=+3676.259436995" Nov 03 21:49:42.039141 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:42.039106 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-776jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 03 21:49:42.039358 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:42.039177 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" podUID="7803de3d-9e9c-475a-9227-74535c9c1a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 03 21:49:42.058145 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:42.058007 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:42.064243 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:42.064210 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:42.674689 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:42.674653 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:42.675712 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:42.675691 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:49:43.039320 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:43.039280 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-776jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 03 21:49:43.039798 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:43.039345 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" podUID="7803de3d-9e9c-475a-9227-74535c9c1a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 03 21:49:44.211721 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:44.211683 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:44.212633 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:44.212608 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:44.213251 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:44.213226 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 21:49:46.294230 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:46.293963 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6a186d-555fb5d966-t8ppx"] Nov 03 21:49:49.857001 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:49.856966 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt"] Nov 03 21:49:49.861600 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:49.861578 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:49:49.888293 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:49.888255 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mncxt\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:49:49.888477 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:49.888299 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mncxt\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:49:49.888477 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:49.888377 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625d3189-ee5d-4264-b3a2-469ddecb4427-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mncxt\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:49:49.888477 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:49.888417 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc5th\" (UniqueName: \"kubernetes.io/projected/625d3189-ee5d-4264-b3a2-469ddecb4427-kube-api-access-rc5th\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mncxt\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:49:49.888594 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:49.888478 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625d3189-ee5d-4264-b3a2-469ddecb4427-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mncxt\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:49:49.888594 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:49.888560 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625d3189-ee5d-4264-b3a2-469ddecb4427-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mncxt\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:49:49.888664 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:49.888647 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mncxt\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:49:49.888701 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:49.888666 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mncxt\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:49:49.888736 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:49.888714 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mncxt\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:49:49.989884 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:49.989845 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625d3189-ee5d-4264-b3a2-469ddecb4427-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mncxt\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:49:49.989884 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:49.989890 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625d3189-ee5d-4264-b3a2-469ddecb4427-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mncxt\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:49:49.990208 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:49.990097 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mncxt\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:49:49.990208 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:49.990134 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mncxt\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:49:49.990390 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:49.990219 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mncxt\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:49:49.990457 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:49.990434 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mncxt\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:49:49.990514 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:49.990471 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mncxt\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:49:49.990608 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:49.990538 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625d3189-ee5d-4264-b3a2-469ddecb4427-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mncxt\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:49:49.990676 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:49.990615 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc5th\" (UniqueName: \"kubernetes.io/projected/625d3189-ee5d-4264-b3a2-469ddecb4427-kube-api-access-rc5th\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mncxt\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:49:49.990779 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:49.990755 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625d3189-ee5d-4264-b3a2-469ddecb4427-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mncxt\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:49:49.992170 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:49.991456 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mncxt\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:49:49.992170 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:49.991715 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mncxt\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:49:49.992170 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:49.991939 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mncxt\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:49:49.992602 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:49.992571 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mncxt\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:49:49.994332 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:49.994284 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mncxt\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:49:49.994833 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:49.994808 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625d3189-ee5d-4264-b3a2-469ddecb4427-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mncxt\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:49:50.053241 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:50.053208 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625d3189-ee5d-4264-b3a2-469ddecb4427-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mncxt\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:49:50.053424 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:50.053307 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc5th\" (UniqueName: \"kubernetes.io/projected/625d3189-ee5d-4264-b3a2-469ddecb4427-kube-api-access-rc5th\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mncxt\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:49:50.068386 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:50.068352 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt"] Nov 03 21:49:50.177376 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:50.177290 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:49:50.614658 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:49:50.614618 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625d3189_ee5d_4264_b3a2_469ddecb4427.slice/crio-2aa4131b174cf6f3f0d15177deb1762b8490199bf847e4c2ed7b760107b39d10 WatchSource:0}: Error finding container 2aa4131b174cf6f3f0d15177deb1762b8490199bf847e4c2ed7b760107b39d10: Status 404 returned error can't find the container with id 2aa4131b174cf6f3f0d15177deb1762b8490199bf847e4c2ed7b760107b39d10 Nov 03 21:49:50.617139 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:50.617111 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt"] Nov 03 21:49:50.617675 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:50.617639 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:49:50.617717 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:50.617709 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:49:50.617749 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:50.617741 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:49:50.912402 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:50.912304 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" event={"ID":"625d3189-ee5d-4264-b3a2-469ddecb4427","Type":"ContainerStarted","Data":"5f359253c9aa77816a535686cd2bbbfd19cff1a7ddc321b65652ed2fd2d4426a"} Nov 03 21:49:50.913023 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:50.912995 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" event={"ID":"625d3189-ee5d-4264-b3a2-469ddecb4427","Type":"ContainerStarted","Data":"2aa4131b174cf6f3f0d15177deb1762b8490199bf847e4c2ed7b760107b39d10"} Nov 03 21:49:51.050656 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:51.050585 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" podStartSLOduration=2.050567587 podStartE2EDuration="2.050567587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:51.048427789 +0000 UTC m=+3685.516206513" watchObservedRunningTime="2026-11-03 21:49:51.050567587 +0000 UTC m=+3685.518346309" Nov 03 21:49:51.178223 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:51.178126 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:49:51.180060 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:51.180021 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-mncxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Nov 03 21:49:51.180250 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:51.180121 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" podUID="625d3189-ee5d-4264-b3a2-469ddecb4427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Nov 03 21:49:51.302031 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:51.301966 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-6a186d-555fb5d966-t8ppx" podUID="12271b60-16d6-4716-8db7-fddd1d5bbd8b" containerName="payload-processing" containerID="cri-o://5594f4c1c6437d6669f454f97e9420e0f52e8024e631cd6c9328fb10a59ee83d" gracePeriod=25 Nov 03 21:49:51.593242 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:51.593213 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6a186d-555fb5d966-t8ppx" Nov 03 21:49:51.709367 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:51.709326 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12271b60-16d6-4716-8db7-fddd1d5bbd8b-kube-api-access-wgj7q\" (UniqueName: \"kubernetes.io/projected/12271b60-16d6-4716-8db7-fddd1d5bbd8b-kube-api-access-wgj7q\") pod \"12271b60-16d6-4716-8db7-fddd1d5bbd8b\" (UID: \"12271b60-16d6-4716-8db7-fddd1d5bbd8b\") " Nov 03 21:49:51.709599 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:51.709581 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12271b60-16d6-4716-8db7-fddd1d5bbd8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12271b60-16d6-4716-8db7-fddd1d5bbd8b-plugins-config-volume\") pod \"12271b60-16d6-4716-8db7-fddd1d5bbd8b\" (UID: \"12271b60-16d6-4716-8db7-fddd1d5bbd8b\") " Nov 03 21:49:51.709971 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:51.709944 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12271b60-16d6-4716-8db7-fddd1d5bbd8b-plugins-config-volume" pod "12271b60-16d6-4716-8db7-fddd1d5bbd8b" (UID: "12271b60-16d6-4716-8db7-fddd1d5bbd8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:49:51.710144 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:51.710133 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12271b60-16d6-4716-8db7-fddd1d5bbd8b-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:49:51.712000 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:51.711973 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12271b60-16d6-4716-8db7-fddd1d5bbd8b-kube-api-access-wgj7q" pod "12271b60-16d6-4716-8db7-fddd1d5bbd8b" (UID: "12271b60-16d6-4716-8db7-fddd1d5bbd8b"). InnerVolumeSpecName "kube-api-access-wgj7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:49:51.811085 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:51.811048 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgj7q\" (UniqueName: \"kubernetes.io/projected/12271b60-16d6-4716-8db7-fddd1d5bbd8b-kube-api-access-wgj7q\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:49:51.928499 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:51.928463 5615 generic.go:356] "Generic (PLEG): container finished" podID="12271b60-16d6-4716-8db7-fddd1d5bbd8b" containerID="5594f4c1c6437d6669f454f97e9420e0f52e8024e631cd6c9328fb10a59ee83d" exitCode=0 Nov 03 21:49:51.928946 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:51.928518 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6a186d-555fb5d966-t8ppx" Nov 03 21:49:51.928946 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:51.928545 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6a186d-555fb5d966-t8ppx" event={"ID":"12271b60-16d6-4716-8db7-fddd1d5bbd8b","Type":"ContainerDied","Data":"5594f4c1c6437d6669f454f97e9420e0f52e8024e631cd6c9328fb10a59ee83d"} Nov 03 21:49:51.928946 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:51.928593 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6a186d-555fb5d966-t8ppx" event={"ID":"12271b60-16d6-4716-8db7-fddd1d5bbd8b","Type":"ContainerDied","Data":"e500aa193b12c9f53def28f803978e02b2be07fe2aedf917f5a5666ea345edb6"} Nov 03 21:49:51.928946 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:51.928616 5615 scope.go:122] "RemoveContainer" containerID="5594f4c1c6437d6669f454f97e9420e0f52e8024e631cd6c9328fb10a59ee83d" Nov 03 21:49:51.939184 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:51.939144 5615 scope.go:122] "RemoveContainer" containerID="5594f4c1c6437d6669f454f97e9420e0f52e8024e631cd6c9328fb10a59ee83d" Nov 03 21:49:51.939539 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:49:51.939520 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5594f4c1c6437d6669f454f97e9420e0f52e8024e631cd6c9328fb10a59ee83d\": container with ID starting with 5594f4c1c6437d6669f454f97e9420e0f52e8024e631cd6c9328fb10a59ee83d not found: ID does not exist" containerID="5594f4c1c6437d6669f454f97e9420e0f52e8024e631cd6c9328fb10a59ee83d" Nov 03 21:49:51.939589 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:51.939550 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5594f4c1c6437d6669f454f97e9420e0f52e8024e631cd6c9328fb10a59ee83d"} err="failed to get container status \"5594f4c1c6437d6669f454f97e9420e0f52e8024e631cd6c9328fb10a59ee83d\": rpc error: code = NotFound desc = could not find container \"5594f4c1c6437d6669f454f97e9420e0f52e8024e631cd6c9328fb10a59ee83d\": container with ID starting with 5594f4c1c6437d6669f454f97e9420e0f52e8024e631cd6c9328fb10a59ee83d not found: ID does not exist" Nov 03 21:49:52.108522 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:52.108391 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6a186d-555fb5d966-t8ppx"] Nov 03 21:49:52.112006 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:52.111971 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6a186d-555fb5d966-t8ppx"] Nov 03 21:49:52.178228 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:52.178188 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-mncxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Nov 03 21:49:52.178403 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:52.178249 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" podUID="625d3189-ee5d-4264-b3a2-469ddecb4427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Nov 03 21:49:53.177947 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:53.177910 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-mncxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Nov 03 21:49:53.178355 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:53.177964 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" podUID="625d3189-ee5d-4264-b3a2-469ddecb4427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Nov 03 21:49:53.653298 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:53.653264 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12271b60-16d6-4716-8db7-fddd1d5bbd8b" path="/var/lib/kubelet/pods/12271b60-16d6-4716-8db7-fddd1d5bbd8b/volumes" Nov 03 21:49:54.178512 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:54.178475 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-mncxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Nov 03 21:49:54.178929 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:54.178529 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" podUID="625d3189-ee5d-4264-b3a2-469ddecb4427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Nov 03 21:49:55.177862 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:55.177826 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-mncxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Nov 03 21:49:55.178034 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:55.177887 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" podUID="625d3189-ee5d-4264-b3a2-469ddecb4427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Nov 03 21:49:56.178701 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:56.178650 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-mncxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Nov 03 21:49:56.179161 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:56.178718 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" podUID="625d3189-ee5d-4264-b3a2-469ddecb4427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Nov 03 21:49:57.177945 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:57.177902 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-mncxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Nov 03 21:49:57.178178 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:57.177974 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" podUID="625d3189-ee5d-4264-b3a2-469ddecb4427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Nov 03 21:49:58.178371 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:58.178333 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-mncxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Nov 03 21:49:58.178794 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:49:58.178389 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" podUID="625d3189-ee5d-4264-b3a2-469ddecb4427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Nov 03 21:50:00.178731 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:00.178682 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-mncxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:50:00.179225 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:00.178781 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" podUID="625d3189-ee5d-4264-b3a2-469ddecb4427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:50:01.179765 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:01.179710 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-mncxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:50:01.179765 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:01.179769 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" podUID="625d3189-ee5d-4264-b3a2-469ddecb4427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:50:01.180939 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:01.180915 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:50:01.181193 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:01.181174 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:50:01.181954 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:01.181936 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 21:50:05.965154 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:05.964264 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v"] Nov 03 21:50:05.965768 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:05.965352 5615 cpu_manager.go:414] "containerMap: removing container" podUID="12271b60-16d6-4716-8db7-fddd1d5bbd8b" containerName="payload-processing" Nov 03 21:50:05.965768 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:05.965376 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12271b60-16d6-4716-8db7-fddd1d5bbd8b" containerName="payload-processing" Nov 03 21:50:05.965768 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:05.965524 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="12271b60-16d6-4716-8db7-fddd1d5bbd8b" containerName="payload-processing" Nov 03 21:50:05.970651 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:05.969805 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:06.035844 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.031489 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-76eaf8-openshift-default-dockercfg-vg4r4\"" Nov 03 21:50:06.063099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.047298 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38287625-1740-41e8-937c-a7e55e2d8c88-istio-podinfo\") pod \"e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:06.063487 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.063452 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvwpj\" (UniqueName: \"kubernetes.io/projected/38287625-1740-41e8-937c-a7e55e2d8c88-kube-api-access-fvwpj\") pod \"e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:06.081125 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.063665 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-workload-certs\") pod \"e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:06.081514 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.081482 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38287625-1740-41e8-937c-a7e55e2d8c88-istio-token\") pod \"e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:06.081829 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.081803 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-credential-socket\") pod \"e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:06.086098 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.081984 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-workload-socket\") pod \"e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:06.106102 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.086401 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38287625-1740-41e8-937c-a7e55e2d8c88-istiod-ca-cert\") pod \"e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:06.129024 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.128987 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-istio-data\") pod \"e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:06.154229 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.154180 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-istio-envoy\") pod \"e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:06.158294 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.158128 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v"] Nov 03 21:50:06.255725 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.255677 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvwpj\" (UniqueName: \"kubernetes.io/projected/38287625-1740-41e8-937c-a7e55e2d8c88-kube-api-access-fvwpj\") pod \"e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:06.255944 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.255760 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-workload-certs\") pod \"e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:06.255944 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.255805 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38287625-1740-41e8-937c-a7e55e2d8c88-istio-token\") pod \"e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:06.255944 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.255937 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-credential-socket\") pod \"e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:06.256145 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.255987 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-workload-socket\") pod \"e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:06.256145 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.256087 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38287625-1740-41e8-937c-a7e55e2d8c88-istiod-ca-cert\") pod \"e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:06.256145 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.256124 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-istio-data\") pod \"e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:06.256145 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.256140 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-istio-envoy\") pod \"e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:06.256382 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.256177 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38287625-1740-41e8-937c-a7e55e2d8c88-istio-podinfo\") pod \"e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:06.256382 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.256240 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-workload-certs\") pod \"e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:06.256482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.256375 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-credential-socket\") pod \"e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:06.256482 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.256409 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-workload-socket\") pod \"e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:06.256562 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.256542 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-istio-data\") pod \"e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:06.256779 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.256755 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38287625-1740-41e8-937c-a7e55e2d8c88-istiod-ca-cert\") pod \"e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:06.258396 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.258372 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-istio-envoy\") pod \"e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:06.258689 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.258669 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38287625-1740-41e8-937c-a7e55e2d8c88-istio-podinfo\") pod \"e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:06.324951 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.324880 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvwpj\" (UniqueName: \"kubernetes.io/projected/38287625-1740-41e8-937c-a7e55e2d8c88-kube-api-access-fvwpj\") pod \"e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:06.325287 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.325236 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38287625-1740-41e8-937c-a7e55e2d8c88-istio-token\") pod \"e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:06.465006 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.464957 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:06.892982 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:50:06.892951 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38287625_1740_41e8_937c_a7e55e2d8c88.slice/crio-82f0f5b01993778cce9d344deb61c0a2eb9fa5e4bdc42015fd8f26712db9bd64 WatchSource:0}: Error finding container 82f0f5b01993778cce9d344deb61c0a2eb9fa5e4bdc42015fd8f26712db9bd64: Status 404 returned error can't find the container with id 82f0f5b01993778cce9d344deb61c0a2eb9fa5e4bdc42015fd8f26712db9bd64 Nov 03 21:50:06.895975 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.895951 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v"] Nov 03 21:50:06.896228 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.896196 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:50:06.896278 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.896259 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:50:06.896320 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:06.896289 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:50:07.122756 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:07.122712 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" event={"ID":"38287625-1740-41e8-937c-a7e55e2d8c88","Type":"ContainerStarted","Data":"66fb649ee98dd0caf029761a3bd756229e952475fe89f672b7c105ff83cfae24"} Nov 03 21:50:07.123341 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:07.122770 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" event={"ID":"38287625-1740-41e8-937c-a7e55e2d8c88","Type":"ContainerStarted","Data":"82f0f5b01993778cce9d344deb61c0a2eb9fa5e4bdc42015fd8f26712db9bd64"} Nov 03 21:50:07.247342 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:07.247281 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" podStartSLOduration=2.247267355 podStartE2EDuration="2.247267355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:50:07.244909476 +0000 UTC m=+3701.712688198" watchObservedRunningTime="2026-11-03 21:50:07.247267355 +0000 UTC m=+3701.715046077" Nov 03 21:50:07.465158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:07.465121 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:07.471433 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:07.471406 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:08.129543 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:08.129506 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:08.130530 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:50:08.130501 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:50:10.620839 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:50:10.620804 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:50:40.619273 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:50:40.619235 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:51:03.533374 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:03.533338 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76"] Nov 03 21:51:03.533795 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:03.533539 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" podUID="4615ffcf-3653-436d-bfc4-ecda29430be4" containerName="istio-proxy" containerID="cri-o://f00c6fc7d412af44722d22ce3c6da10f1ae3263774686edffd74405cf640a33d" gracePeriod=30 Nov 03 21:51:08.800692 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:08.800627 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:51:08.839722 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:08.839673 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4615ffcf-3653-436d-bfc4-ecda29430be4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4615ffcf-3653-436d-bfc4-ecda29430be4-istiod-ca-cert\") pod \"4615ffcf-3653-436d-bfc4-ecda29430be4\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " Nov 03 21:51:08.839934 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:08.839738 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-workload-socket\") pod \"4615ffcf-3653-436d-bfc4-ecda29430be4\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " Nov 03 21:51:08.839934 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:08.839797 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-istio-envoy\") pod \"4615ffcf-3653-436d-bfc4-ecda29430be4\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " Nov 03 21:51:08.840054 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:08.839969 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-workload-certs\") pod \"4615ffcf-3653-436d-bfc4-ecda29430be4\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " Nov 03 21:51:08.840054 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:08.840027 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-workload-socket" pod "4615ffcf-3653-436d-bfc4-ecda29430be4" (UID: "4615ffcf-3653-436d-bfc4-ecda29430be4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:51:08.840054 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:08.840047 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-istio-data\") pod \"4615ffcf-3653-436d-bfc4-ecda29430be4\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " Nov 03 21:51:08.840247 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:08.840118 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-credential-socket\") pod \"4615ffcf-3653-436d-bfc4-ecda29430be4\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " Nov 03 21:51:08.840247 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:08.840154 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4615ffcf-3653-436d-bfc4-ecda29430be4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4615ffcf-3653-436d-bfc4-ecda29430be4-istio-podinfo\") pod \"4615ffcf-3653-436d-bfc4-ecda29430be4\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " Nov 03 21:51:08.840247 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:08.840211 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4615ffcf-3653-436d-bfc4-ecda29430be4-kube-api-access-llj4x\" (UniqueName: \"kubernetes.io/projected/4615ffcf-3653-436d-bfc4-ecda29430be4-kube-api-access-llj4x\") pod \"4615ffcf-3653-436d-bfc4-ecda29430be4\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " Nov 03 21:51:08.840402 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:08.840254 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-workload-certs" pod "4615ffcf-3653-436d-bfc4-ecda29430be4" (UID: "4615ffcf-3653-436d-bfc4-ecda29430be4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:51:08.840402 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:08.840273 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-istio-data" pod "4615ffcf-3653-436d-bfc4-ecda29430be4" (UID: "4615ffcf-3653-436d-bfc4-ecda29430be4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:51:08.840402 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:08.840180 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4615ffcf-3653-436d-bfc4-ecda29430be4-istiod-ca-cert" pod "4615ffcf-3653-436d-bfc4-ecda29430be4" (UID: "4615ffcf-3653-436d-bfc4-ecda29430be4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:51:08.840402 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:08.840276 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4615ffcf-3653-436d-bfc4-ecda29430be4-istio-token\" (UniqueName: \"kubernetes.io/projected/4615ffcf-3653-436d-bfc4-ecda29430be4-istio-token\") pod \"4615ffcf-3653-436d-bfc4-ecda29430be4\" (UID: \"4615ffcf-3653-436d-bfc4-ecda29430be4\") " Nov 03 21:51:08.840776 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:08.840750 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-credential-socket" pod "4615ffcf-3653-436d-bfc4-ecda29430be4" (UID: "4615ffcf-3653-436d-bfc4-ecda29430be4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:51:08.841134 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:08.840978 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:51:08.841134 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:08.841001 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:51:08.841134 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:08.841018 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:51:08.841134 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:08.841031 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4615ffcf-3653-436d-bfc4-ecda29430be4-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:51:08.841134 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:08.841045 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:51:08.842641 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:08.842614 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4615ffcf-3653-436d-bfc4-ecda29430be4-istio-token" pod "4615ffcf-3653-436d-bfc4-ecda29430be4" (UID: "4615ffcf-3653-436d-bfc4-ecda29430be4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:51:08.842761 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:08.842646 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-istio-envoy" pod "4615ffcf-3653-436d-bfc4-ecda29430be4" (UID: "4615ffcf-3653-436d-bfc4-ecda29430be4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:51:08.842814 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:08.842798 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4615ffcf-3653-436d-bfc4-ecda29430be4-kube-api-access-llj4x" pod "4615ffcf-3653-436d-bfc4-ecda29430be4" (UID: "4615ffcf-3653-436d-bfc4-ecda29430be4"). InnerVolumeSpecName "kube-api-access-llj4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:51:08.842891 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:08.842873 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4615ffcf-3653-436d-bfc4-ecda29430be4-istio-podinfo" pod "4615ffcf-3653-436d-bfc4-ecda29430be4" (UID: "4615ffcf-3653-436d-bfc4-ecda29430be4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:51:08.941883 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:08.941841 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4615ffcf-3653-436d-bfc4-ecda29430be4-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:51:08.941883 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:08.941874 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4615ffcf-3653-436d-bfc4-ecda29430be4-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:51:08.941883 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:08.941886 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llj4x\" (UniqueName: \"kubernetes.io/projected/4615ffcf-3653-436d-bfc4-ecda29430be4-kube-api-access-llj4x\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:51:08.941883 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:08.941895 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4615ffcf-3653-436d-bfc4-ecda29430be4-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:51:09.561798 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:09.561757 5615 generic.go:356] "Generic (PLEG): container finished" podID="4615ffcf-3653-436d-bfc4-ecda29430be4" containerID="f00c6fc7d412af44722d22ce3c6da10f1ae3263774686edffd74405cf640a33d" exitCode=0 Nov 03 21:51:09.561984 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:09.561819 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" Nov 03 21:51:09.561984 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:09.561848 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" event={"ID":"4615ffcf-3653-436d-bfc4-ecda29430be4","Type":"ContainerDied","Data":"f00c6fc7d412af44722d22ce3c6da10f1ae3263774686edffd74405cf640a33d"} Nov 03 21:51:09.561984 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:09.561935 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76" event={"ID":"4615ffcf-3653-436d-bfc4-ecda29430be4","Type":"ContainerDied","Data":"3921dd03d0276af4e5ac301d1696a4d2dc61d79a2f1e1cf1468d809ed9110961"} Nov 03 21:51:09.561984 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:09.561963 5615 scope.go:122] "RemoveContainer" containerID="f00c6fc7d412af44722d22ce3c6da10f1ae3263774686edffd74405cf640a33d" Nov 03 21:51:09.577181 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:09.577151 5615 scope.go:122] "RemoveContainer" containerID="afa3e57ac5ee5015b83f3fa5cfa7dd3243592976bed789f6ae5eebc2c7787bf1" Nov 03 21:51:09.588120 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:09.588070 5615 scope.go:122] "RemoveContainer" containerID="f00c6fc7d412af44722d22ce3c6da10f1ae3263774686edffd74405cf640a33d" Nov 03 21:51:09.588422 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:51:09.588402 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f00c6fc7d412af44722d22ce3c6da10f1ae3263774686edffd74405cf640a33d\": container with ID starting with f00c6fc7d412af44722d22ce3c6da10f1ae3263774686edffd74405cf640a33d not found: ID does not exist" containerID="f00c6fc7d412af44722d22ce3c6da10f1ae3263774686edffd74405cf640a33d" Nov 03 21:51:09.588494 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:09.588431 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f00c6fc7d412af44722d22ce3c6da10f1ae3263774686edffd74405cf640a33d"} err="failed to get container status \"f00c6fc7d412af44722d22ce3c6da10f1ae3263774686edffd74405cf640a33d\": rpc error: code = NotFound desc = could not find container \"f00c6fc7d412af44722d22ce3c6da10f1ae3263774686edffd74405cf640a33d\": container with ID starting with f00c6fc7d412af44722d22ce3c6da10f1ae3263774686edffd74405cf640a33d not found: ID does not exist" Nov 03 21:51:09.588494 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:09.588451 5615 scope.go:122] "RemoveContainer" containerID="afa3e57ac5ee5015b83f3fa5cfa7dd3243592976bed789f6ae5eebc2c7787bf1" Nov 03 21:51:09.588691 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:51:09.588676 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afa3e57ac5ee5015b83f3fa5cfa7dd3243592976bed789f6ae5eebc2c7787bf1\": container with ID starting with afa3e57ac5ee5015b83f3fa5cfa7dd3243592976bed789f6ae5eebc2c7787bf1 not found: ID does not exist" containerID="afa3e57ac5ee5015b83f3fa5cfa7dd3243592976bed789f6ae5eebc2c7787bf1" Nov 03 21:51:09.588731 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:09.588694 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afa3e57ac5ee5015b83f3fa5cfa7dd3243592976bed789f6ae5eebc2c7787bf1"} err="failed to get container status \"afa3e57ac5ee5015b83f3fa5cfa7dd3243592976bed789f6ae5eebc2c7787bf1\": rpc error: code = NotFound desc = could not find container \"afa3e57ac5ee5015b83f3fa5cfa7dd3243592976bed789f6ae5eebc2c7787bf1\": container with ID starting with afa3e57ac5ee5015b83f3fa5cfa7dd3243592976bed789f6ae5eebc2c7787bf1 not found: ID does not exist" Nov 03 21:51:09.788551 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:09.788504 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76"] Nov 03 21:51:09.839104 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:09.839014 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-6a186d-openshift-default-7dbd65f75c-dqb76"] Nov 03 21:51:10.620215 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:51:10.620173 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:51:11.655045 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:11.655006 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4615ffcf-3653-436d-bfc4-ecda29430be4" path="/var/lib/kubelet/pods/4615ffcf-3653-436d-bfc4-ecda29430be4/volumes" Nov 03 21:51:12.756198 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:12.756143 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9e9142-7f8b6dbf5f-5vq49"] Nov 03 21:51:12.765099 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:12.757172 5615 cpu_manager.go:414] "containerMap: removing container" podUID="4615ffcf-3653-436d-bfc4-ecda29430be4" containerName="istio-proxy" Nov 03 21:51:12.765359 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:12.765335 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4615ffcf-3653-436d-bfc4-ecda29430be4" containerName="istio-proxy" Nov 03 21:51:12.765478 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:12.765468 5615 cpu_manager.go:414] "containerMap: removing container" podUID="4615ffcf-3653-436d-bfc4-ecda29430be4" containerName="istio-proxy" Nov 03 21:51:12.765552 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:12.765543 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4615ffcf-3653-436d-bfc4-ecda29430be4" containerName="istio-proxy" Nov 03 21:51:12.765860 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:12.765842 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="4615ffcf-3653-436d-bfc4-ecda29430be4" containerName="istio-proxy" Nov 03 21:51:12.787100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:12.785645 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e9142-7f8b6dbf5f-5vq49" Nov 03 21:51:12.806395 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:12.806359 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9e9142-7f8b6dbf5f-5vq49"] Nov 03 21:51:12.808637 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:12.808608 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb102e63-94c5-4eed-ad81-dbb253fbc44d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9e9142-7f8b6dbf5f-5vq49\" (UID: \"fb102e63-94c5-4eed-ad81-dbb253fbc44d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e9142-7f8b6dbf5f-5vq49" Nov 03 21:51:12.808822 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:12.808801 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnc5w\" (UniqueName: \"kubernetes.io/projected/fb102e63-94c5-4eed-ad81-dbb253fbc44d-kube-api-access-fnc5w\") pod \"payload-pre-processing-e2e-inf-a-9e9142-7f8b6dbf5f-5vq49\" (UID: \"fb102e63-94c5-4eed-ad81-dbb253fbc44d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e9142-7f8b6dbf5f-5vq49" Nov 03 21:51:12.898109 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:12.896623 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9e9142\"" Nov 03 21:51:12.898109 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:12.896790 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-9e9142-dockercfg-k78qm\"" Nov 03 21:51:12.914101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:12.910860 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb102e63-94c5-4eed-ad81-dbb253fbc44d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9e9142-7f8b6dbf5f-5vq49\" (UID: \"fb102e63-94c5-4eed-ad81-dbb253fbc44d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e9142-7f8b6dbf5f-5vq49" Nov 03 21:51:12.914101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:12.911039 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnc5w\" (UniqueName: \"kubernetes.io/projected/fb102e63-94c5-4eed-ad81-dbb253fbc44d-kube-api-access-fnc5w\") pod \"payload-pre-processing-e2e-inf-a-9e9142-7f8b6dbf5f-5vq49\" (UID: \"fb102e63-94c5-4eed-ad81-dbb253fbc44d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e9142-7f8b6dbf5f-5vq49" Nov 03 21:51:12.933109 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:12.920463 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9e9142-65cfd77f76-x8thm"] Nov 03 21:51:12.933109 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:12.929453 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="4615ffcf-3653-436d-bfc4-ecda29430be4" containerName="istio-proxy" Nov 03 21:51:12.950101 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:12.948660 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb102e63-94c5-4eed-ad81-dbb253fbc44d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9e9142-7f8b6dbf5f-5vq49\" (UID: \"fb102e63-94c5-4eed-ad81-dbb253fbc44d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e9142-7f8b6dbf5f-5vq49" Nov 03 21:51:12.968599 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:12.968563 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9e9142-65cfd77f76-x8thm" Nov 03 21:51:12.984170 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:12.984059 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnc5w\" (UniqueName: \"kubernetes.io/projected/fb102e63-94c5-4eed-ad81-dbb253fbc44d-kube-api-access-fnc5w\") pod \"payload-pre-processing-e2e-inf-a-9e9142-7f8b6dbf5f-5vq49\" (UID: \"fb102e63-94c5-4eed-ad81-dbb253fbc44d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e9142-7f8b6dbf5f-5vq49" Nov 03 21:51:13.009721 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:13.009589 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9e9142-65cfd77f76-x8thm"] Nov 03 21:51:13.012413 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:13.012382 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b18bd0d-0a30-4471-8ced-27cce9e7b5b0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9e9142-65cfd77f76-x8thm\" (UID: \"4b18bd0d-0a30-4471-8ced-27cce9e7b5b0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9e9142-65cfd77f76-x8thm" Nov 03 21:51:13.012779 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:13.012624 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njgq9\" (UniqueName: \"kubernetes.io/projected/4b18bd0d-0a30-4471-8ced-27cce9e7b5b0-kube-api-access-njgq9\") pod \"payload-processing-e2e-inf-a-9e9142-65cfd77f76-x8thm\" (UID: \"4b18bd0d-0a30-4471-8ced-27cce9e7b5b0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9e9142-65cfd77f76-x8thm" Nov 03 21:51:13.109997 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:13.109964 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e9142-7f8b6dbf5f-5vq49" Nov 03 21:51:13.114766 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:13.114733 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b18bd0d-0a30-4471-8ced-27cce9e7b5b0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9e9142-65cfd77f76-x8thm\" (UID: \"4b18bd0d-0a30-4471-8ced-27cce9e7b5b0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9e9142-65cfd77f76-x8thm" Nov 03 21:51:13.114855 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:13.114808 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njgq9\" (UniqueName: \"kubernetes.io/projected/4b18bd0d-0a30-4471-8ced-27cce9e7b5b0-kube-api-access-njgq9\") pod \"payload-processing-e2e-inf-a-9e9142-65cfd77f76-x8thm\" (UID: \"4b18bd0d-0a30-4471-8ced-27cce9e7b5b0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9e9142-65cfd77f76-x8thm" Nov 03 21:51:13.115560 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:13.115531 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b18bd0d-0a30-4471-8ced-27cce9e7b5b0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9e9142-65cfd77f76-x8thm\" (UID: \"4b18bd0d-0a30-4471-8ced-27cce9e7b5b0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9e9142-65cfd77f76-x8thm" Nov 03 21:51:13.177846 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:13.177802 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njgq9\" (UniqueName: \"kubernetes.io/projected/4b18bd0d-0a30-4471-8ced-27cce9e7b5b0-kube-api-access-njgq9\") pod \"payload-processing-e2e-inf-a-9e9142-65cfd77f76-x8thm\" (UID: \"4b18bd0d-0a30-4471-8ced-27cce9e7b5b0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9e9142-65cfd77f76-x8thm" Nov 03 21:51:13.320386 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:13.320302 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9e9142-65cfd77f76-x8thm" Nov 03 21:51:13.516158 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:51:13.516128 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb102e63_94c5_4eed_ad81_dbb253fbc44d.slice/crio-52360170ba37b29af53796441faf7e0645dc85048781003cd6247f885dadccd5 WatchSource:0}: Error finding container 52360170ba37b29af53796441faf7e0645dc85048781003cd6247f885dadccd5: Status 404 returned error can't find the container with id 52360170ba37b29af53796441faf7e0645dc85048781003cd6247f885dadccd5 Nov 03 21:51:13.518522 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:13.518498 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9e9142-7f8b6dbf5f-5vq49"] Nov 03 21:51:13.599951 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:13.599914 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e9142-7f8b6dbf5f-5vq49" event={"ID":"fb102e63-94c5-4eed-ad81-dbb253fbc44d","Type":"ContainerStarted","Data":"52360170ba37b29af53796441faf7e0645dc85048781003cd6247f885dadccd5"} Nov 03 21:51:13.697647 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:51:13.697610 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b18bd0d_0a30_4471_8ced_27cce9e7b5b0.slice/crio-17ff6dccd4bca0816d9eb12b0e62c12ff453cfc53fb614bfe38ea3d606534f98 WatchSource:0}: Error finding container 17ff6dccd4bca0816d9eb12b0e62c12ff453cfc53fb614bfe38ea3d606534f98: Status 404 returned error can't find the container with id 17ff6dccd4bca0816d9eb12b0e62c12ff453cfc53fb614bfe38ea3d606534f98 Nov 03 21:51:13.702017 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:13.701986 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9e9142-65cfd77f76-x8thm"] Nov 03 21:51:14.608459 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:14.608419 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9e9142-65cfd77f76-x8thm" event={"ID":"4b18bd0d-0a30-4471-8ced-27cce9e7b5b0","Type":"ContainerStarted","Data":"3a1085efe9ef68d5e27e54a6e3e0e864744f9faea71624df6705bb0a3c9f9ec5"} Nov 03 21:51:14.608961 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:14.608472 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9e9142-65cfd77f76-x8thm" event={"ID":"4b18bd0d-0a30-4471-8ced-27cce9e7b5b0","Type":"ContainerStarted","Data":"17ff6dccd4bca0816d9eb12b0e62c12ff453cfc53fb614bfe38ea3d606534f98"} Nov 03 21:51:14.608961 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:14.608525 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9e9142-65cfd77f76-x8thm" Nov 03 21:51:14.610229 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:14.610205 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e9142-7f8b6dbf5f-5vq49" event={"ID":"fb102e63-94c5-4eed-ad81-dbb253fbc44d","Type":"ContainerStarted","Data":"0e271f3643e8ccb87a91e77d250e2fdb018e38d06c2539f473131b52a5c9160d"} Nov 03 21:51:14.610340 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:14.610247 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e9142-7f8b6dbf5f-5vq49" Nov 03 21:51:14.726410 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:14.726361 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-9e9142-65cfd77f76-x8thm" podStartSLOduration=2.7263491650000002 podStartE2EDuration="2.726349165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:51:14.72588385 +0000 UTC m=+3769.193662572" watchObservedRunningTime="2026-11-03 21:51:14.726349165 +0000 UTC m=+3769.194127887" Nov 03 21:51:25.619542 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:25.619511 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e9142-7f8b6dbf5f-5vq49" Nov 03 21:51:25.619948 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:25.619581 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9e9142-65cfd77f76-x8thm" Nov 03 21:51:25.739746 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:25.739687 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e9142-7f8b6dbf5f-5vq49" podStartSLOduration=13.739672803 podStartE2EDuration="13.739672803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:51:14.861364389 +0000 UTC m=+3769.329143113" watchObservedRunningTime="2026-11-03 21:51:25.739672803 +0000 UTC m=+3780.207451577" Nov 03 21:51:40.621070 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:51:40.621023 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:51:43.156798 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:43.154665 5615 patch_prober.go:29] interesting pod/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": read tcp 10.134.0.2:47828->10.134.0.181:15021: read: connection reset by peer" start-of-body= Nov 03 21:51:43.156798 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:43.154736 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" podUID="0f2fd230-b1c2-4835-bc30-25437b149c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": read tcp 10.134.0.2:47828->10.134.0.181:15021: read: connection reset by peer" Nov 03 21:51:43.798404 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:43.798370 5615 generic.go:356] "Generic (PLEG): container finished" podID="0f2fd230-b1c2-4835-bc30-25437b149c02" containerID="858c66df7ca402d81c7079c5df4851c1c0b37b2f543db77f6cc21bdd9eef5cc7" exitCode=0 Nov 03 21:51:43.798584 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:43.798442 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" event={"ID":"0f2fd230-b1c2-4835-bc30-25437b149c02","Type":"ContainerDied","Data":"858c66df7ca402d81c7079c5df4851c1c0b37b2f543db77f6cc21bdd9eef5cc7"} Nov 03 21:51:43.798883 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:43.798868 5615 scope.go:122] "RemoveContainer" containerID="858c66df7ca402d81c7079c5df4851c1c0b37b2f543db77f6cc21bdd9eef5cc7" Nov 03 21:51:43.800567 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:43.800533 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:51:43.800683 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:43.800618 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:51:43.800683 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:43.800671 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:51:44.831746 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:44.831708 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" event={"ID":"0f2fd230-b1c2-4835-bc30-25437b149c02","Type":"ContainerStarted","Data":"d69b51f960d1deb1d8a5d2199d451990cf1d9a032ceede885b1af47b0709e139"} Nov 03 21:51:45.057719 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:45.057676 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:51:45.057928 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:45.057886 5615 patch_prober.go:29] interesting pod/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 03 21:51:45.058011 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:45.057951 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" podUID="0f2fd230-b1c2-4835-bc30-25437b149c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 03 21:51:46.058107 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:46.058035 5615 patch_prober.go:29] interesting pod/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 03 21:51:46.058491 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:46.058118 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" podUID="0f2fd230-b1c2-4835-bc30-25437b149c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 03 21:51:47.058025 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:47.057989 5615 patch_prober.go:29] interesting pod/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 03 21:51:47.058247 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:47.058044 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" podUID="0f2fd230-b1c2-4835-bc30-25437b149c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 03 21:51:48.058857 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:48.058812 5615 patch_prober.go:29] interesting pod/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 03 21:51:48.059315 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:48.058880 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" podUID="0f2fd230-b1c2-4835-bc30-25437b149c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 03 21:51:49.058290 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:49.058251 5615 patch_prober.go:29] interesting pod/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 03 21:51:49.058489 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:49.058311 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" podUID="0f2fd230-b1c2-4835-bc30-25437b149c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 03 21:51:50.142575 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:50.142534 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:51:50.143118 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:50.143067 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:51:50.144046 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:50.144023 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:51:56.910516 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:56.910480 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7"] Nov 03 21:51:56.914462 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:56.914433 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:51:57.000103 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:56.991506 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0a0da1-openshift-default-dockercfg-hr7sv\"" Nov 03 21:51:57.029100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:56.964069 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45v5t\" (UniqueName: \"kubernetes.io/projected/1ac55cb2-e271-4044-ab8e-cc5a6335af53-kube-api-access-45v5t\") pod \"e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:51:57.037105 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.029427 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istio-data\") pod \"e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:51:57.060260 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.060211 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-workload-certs\") pod \"e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:51:57.060527 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.060509 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istiod-ca-cert\") pod \"e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:51:57.060775 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.060658 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-workload-socket\") pod \"e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:51:57.060930 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.060913 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istio-token\") pod \"e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:51:57.085037 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.068281 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-credential-socket\") pod \"e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:51:57.085464 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.085418 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istio-envoy\") pod \"e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:51:57.085798 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.085769 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istio-podinfo\") pod \"e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:51:57.213669 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.213562 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7"] Nov 03 21:51:57.247480 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.247438 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45v5t\" (UniqueName: \"kubernetes.io/projected/1ac55cb2-e271-4044-ab8e-cc5a6335af53-kube-api-access-45v5t\") pod \"e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:51:57.247685 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.247521 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istio-data\") pod \"e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:51:57.247685 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.247551 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-workload-certs\") pod \"e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:51:57.247685 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.247599 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istiod-ca-cert\") pod \"e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:51:57.247882 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.247686 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-workload-socket\") pod \"e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:51:57.248089 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.248046 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istio-data\") pod \"e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:51:57.248178 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.248119 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-workload-certs\") pod \"e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:51:57.248312 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.248285 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-workload-socket\") pod \"e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:51:57.248478 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.248413 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istio-token\") pod \"e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:51:57.248651 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.248431 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istiod-ca-cert\") pod \"e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:51:57.248808 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.248786 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-credential-socket\") pod \"e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:51:57.248852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.248830 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istio-envoy\") pod \"e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:51:57.248915 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.248900 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istio-podinfo\") pod \"e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:51:57.249133 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.249109 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-credential-socket\") pod \"e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:51:57.251322 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.251287 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istio-envoy\") pod \"e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:51:57.251676 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.251650 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istio-podinfo\") pod \"e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:51:57.320638 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.319677 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istio-token\") pod \"e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:51:57.321407 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.321376 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45v5t\" (UniqueName: \"kubernetes.io/projected/1ac55cb2-e271-4044-ab8e-cc5a6335af53-kube-api-access-45v5t\") pod \"e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:51:57.535337 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.535286 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:51:57.960621 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:51:57.960590 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac55cb2_e271_4044_ab8e_cc5a6335af53.slice/crio-3bf02e1c4a42ba73d8a8a19b60db7721b5d4deac7bbede55eb01c2997717b3c7 WatchSource:0}: Error finding container 3bf02e1c4a42ba73d8a8a19b60db7721b5d4deac7bbede55eb01c2997717b3c7: Status 404 returned error can't find the container with id 3bf02e1c4a42ba73d8a8a19b60db7721b5d4deac7bbede55eb01c2997717b3c7 Nov 03 21:51:57.962351 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.962233 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7"] Nov 03 21:51:57.963405 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.963369 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:51:57.963491 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.963436 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:51:57.963491 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:57.963465 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:51:58.047610 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:58.047560 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" event={"ID":"1ac55cb2-e271-4044-ab8e-cc5a6335af53","Type":"ContainerStarted","Data":"3bf02e1c4a42ba73d8a8a19b60db7721b5d4deac7bbede55eb01c2997717b3c7"} Nov 03 21:51:59.055875 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:59.055834 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" event={"ID":"1ac55cb2-e271-4044-ab8e-cc5a6335af53","Type":"ContainerStarted","Data":"b2188484e61d2824f9553f15dcf77ef8c998b440f86e4f0b001b6b5e3e913518"} Nov 03 21:51:59.268402 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:59.268344 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" podStartSLOduration=3.268329054 podStartE2EDuration="3.268329054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:51:59.240878174 +0000 UTC m=+3813.708656896" watchObservedRunningTime="2026-11-03 21:51:59.268329054 +0000 UTC m=+3813.736107777" Nov 03 21:51:59.536163 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:59.536117 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:51:59.543115 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:51:59.543050 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:52:00.062779 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:00.062741 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:52:00.063776 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:00.063755 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:52:08.121211 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:08.121174 5615 generic.go:356] "Generic (PLEG): container finished" podID="38287625-1740-41e8-937c-a7e55e2d8c88" containerID="66fb649ee98dd0caf029761a3bd756229e952475fe89f672b7c105ff83cfae24" exitCode=0 Nov 03 21:52:08.121793 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:08.121207 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" event={"ID":"38287625-1740-41e8-937c-a7e55e2d8c88","Type":"ContainerDied","Data":"66fb649ee98dd0caf029761a3bd756229e952475fe89f672b7c105ff83cfae24"} Nov 03 21:52:08.121793 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:08.121757 5615 scope.go:122] "RemoveContainer" containerID="66fb649ee98dd0caf029761a3bd756229e952475fe89f672b7c105ff83cfae24" Nov 03 21:52:08.123912 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:08.123873 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:52:08.124024 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:08.123983 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:52:08.124106 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:08.124030 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:52:08.129490 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:08.129465 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:52:09.134288 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:09.134253 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" event={"ID":"38287625-1740-41e8-937c-a7e55e2d8c88","Type":"ContainerStarted","Data":"12107666eae12d0141514ca5553b938284188c16b93fc20d97fe15e682f8cdb7"} Nov 03 21:52:09.466242 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:09.466148 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:52:09.467459 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:09.467432 5615 patch_prober.go:29] interesting pod/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Nov 03 21:52:09.467589 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:09.467485 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" podUID="38287625-1740-41e8-937c-a7e55e2d8c88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Nov 03 21:52:10.466446 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:10.466404 5615 patch_prober.go:29] interesting pod/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Nov 03 21:52:10.466893 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:10.466463 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" podUID="38287625-1740-41e8-937c-a7e55e2d8c88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Nov 03 21:52:10.644594 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:52:10.644553 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:52:11.466254 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:11.466218 5615 patch_prober.go:29] interesting pod/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Nov 03 21:52:11.466461 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:11.466275 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" podUID="38287625-1740-41e8-937c-a7e55e2d8c88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Nov 03 21:52:12.465397 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:12.465359 5615 patch_prober.go:29] interesting pod/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Nov 03 21:52:12.465861 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:12.465425 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" podUID="38287625-1740-41e8-937c-a7e55e2d8c88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Nov 03 21:52:13.465490 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:13.465458 5615 patch_prober.go:29] interesting pod/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Nov 03 21:52:13.465943 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:13.465516 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" podUID="38287625-1740-41e8-937c-a7e55e2d8c88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Nov 03 21:52:14.478582 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:14.478543 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:52:14.479557 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:14.479533 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:52:14.480540 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:14.480510 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:52:40.619653 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:52:40.619613 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:52:42.027611 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:42.027564 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh"] Nov 03 21:52:42.031529 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:42.031507 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh" Nov 03 21:52:42.086015 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:42.085977 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-9e9142-serving-cert\"" Nov 03 21:52:42.107303 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:42.107260 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbc63d42-9000-4c29-b2ff-2c4ef820b337-maas-api-tls\") pod \"maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh\" (UID: \"fbc63d42-9000-4c29-b2ff-2c4ef820b337\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh" Nov 03 21:52:42.107303 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:42.107312 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc63d42-9000-4c29-b2ff-2c4ef820b337-metrics-certs\") pod \"maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh\" (UID: \"fbc63d42-9000-4c29-b2ff-2c4ef820b337\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh" Nov 03 21:52:42.107590 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:42.107430 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5bb4\" (UniqueName: \"kubernetes.io/projected/fbc63d42-9000-4c29-b2ff-2c4ef820b337-kube-api-access-k5bb4\") pod \"maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh\" (UID: \"fbc63d42-9000-4c29-b2ff-2c4ef820b337\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh" Nov 03 21:52:42.107702 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:42.107676 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbc63d42-9000-4c29-b2ff-2c4ef820b337-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh\" (UID: \"fbc63d42-9000-4c29-b2ff-2c4ef820b337\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh" Nov 03 21:52:42.140001 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:42.139965 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh"] Nov 03 21:52:42.208681 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:42.208640 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbc63d42-9000-4c29-b2ff-2c4ef820b337-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh\" (UID: \"fbc63d42-9000-4c29-b2ff-2c4ef820b337\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh" Nov 03 21:52:42.208894 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:42.208729 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbc63d42-9000-4c29-b2ff-2c4ef820b337-maas-api-tls\") pod \"maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh\" (UID: \"fbc63d42-9000-4c29-b2ff-2c4ef820b337\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh" Nov 03 21:52:42.208894 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:42.208752 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc63d42-9000-4c29-b2ff-2c4ef820b337-metrics-certs\") pod \"maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh\" (UID: \"fbc63d42-9000-4c29-b2ff-2c4ef820b337\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh" Nov 03 21:52:42.208894 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:42.208782 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5bb4\" (UniqueName: \"kubernetes.io/projected/fbc63d42-9000-4c29-b2ff-2c4ef820b337-kube-api-access-k5bb4\") pod \"maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh\" (UID: \"fbc63d42-9000-4c29-b2ff-2c4ef820b337\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh" Nov 03 21:52:42.209112 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:52:42.208931 5615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-serving-cert: secret "maas-api-e2e-inf-a-9e9142-serving-cert" not found Nov 03 21:52:42.209112 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:52:42.208999 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc63d42-9000-4c29-b2ff-2c4ef820b337-maas-api-tls podName:fbc63d42-9000-4c29-b2ff-2c4ef820b337 nodeName:}" failed. No retries permitted until 2026-11-03 21:52:42.708980613 +0000 UTC m=+3857.176759336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fbc63d42-9000-4c29-b2ff-2c4ef820b337-maas-api-tls") pod "maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh" (UID: "fbc63d42-9000-4c29-b2ff-2c4ef820b337") : secret "maas-api-e2e-inf-a-9e9142-serving-cert" not found Nov 03 21:52:42.209465 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:42.209441 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbc63d42-9000-4c29-b2ff-2c4ef820b337-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh\" (UID: \"fbc63d42-9000-4c29-b2ff-2c4ef820b337\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh" Nov 03 21:52:42.211279 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:42.211261 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc63d42-9000-4c29-b2ff-2c4ef820b337-metrics-certs\") pod \"maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh\" (UID: \"fbc63d42-9000-4c29-b2ff-2c4ef820b337\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh" Nov 03 21:52:42.271751 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:42.271712 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5bb4\" (UniqueName: \"kubernetes.io/projected/fbc63d42-9000-4c29-b2ff-2c4ef820b337-kube-api-access-k5bb4\") pod \"maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh\" (UID: \"fbc63d42-9000-4c29-b2ff-2c4ef820b337\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh" Nov 03 21:52:42.716790 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:42.716744 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbc63d42-9000-4c29-b2ff-2c4ef820b337-maas-api-tls\") pod \"maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh\" (UID: \"fbc63d42-9000-4c29-b2ff-2c4ef820b337\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh" Nov 03 21:52:42.719300 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:42.719276 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbc63d42-9000-4c29-b2ff-2c4ef820b337-maas-api-tls\") pod \"maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh\" (UID: \"fbc63d42-9000-4c29-b2ff-2c4ef820b337\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh" Nov 03 21:52:42.945711 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:42.945664 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh" Nov 03 21:52:43.323572 ip-10-0-129-192 kubenswrapper[5615]: W1103 21:52:43.323537 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc63d42_9000_4c29_b2ff_2c4ef820b337.slice/crio-fa6d70d7276c5df622f644422b6965bf33a8b33efcee62f58ad536c676982b19 WatchSource:0}: Error finding container fa6d70d7276c5df622f644422b6965bf33a8b33efcee62f58ad536c676982b19: Status 404 returned error can't find the container with id fa6d70d7276c5df622f644422b6965bf33a8b33efcee62f58ad536c676982b19 Nov 03 21:52:43.325442 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:43.325421 5615 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:52:43.325574 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:43.325534 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh"] Nov 03 21:52:43.459707 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:43.459673 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh" event={"ID":"fbc63d42-9000-4c29-b2ff-2c4ef820b337","Type":"ContainerStarted","Data":"fa6d70d7276c5df622f644422b6965bf33a8b33efcee62f58ad536c676982b19"} Nov 03 21:52:44.467620 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:44.467575 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh" event={"ID":"fbc63d42-9000-4c29-b2ff-2c4ef820b337","Type":"ContainerStarted","Data":"1bf865777583c3c9102a0e7b6f64798f27200442f76aa7642790cdc4c9e7ffc3"} Nov 03 21:52:44.468050 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:44.467797 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh" Nov 03 21:52:44.588402 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:44.588304 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh" podStartSLOduration=2.655264771 podStartE2EDuration="3.588290041s" totalImagesPullingTime="933.02527ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:52:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:52:44.58687366 +0000 UTC m=+3859.054652383" watchObservedRunningTime="2026-11-03 21:52:44.588290041 +0000 UTC m=+3859.056068763" Nov 03 21:52:50.482670 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:52:50.482638 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh" Nov 03 21:53:10.620265 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:53:10.620179 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:53:40.658640 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:53:40.658596 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:54:10.648890 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:54:10.648831 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:54:14.959601 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:14.959564 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z"] Nov 03 21:54:14.960276 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:14.959778 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" podUID="0f2fd230-b1c2-4835-bc30-25437b149c02" containerName="istio-proxy" containerID="cri-o://d69b51f960d1deb1d8a5d2199d451990cf1d9a032ceede885b1af47b0709e139" gracePeriod=30 Nov 03 21:54:15.426050 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:15.426005 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9e9142-65cfd77f76-x8thm"] Nov 03 21:54:15.543274 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:15.543128 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9e9142-7f8b6dbf5f-5vq49"] Nov 03 21:54:16.082290 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:16.081485 5615 patch_prober.go:29] interesting pod/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:54:16.082290 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:16.081731 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" podUID="1ac55cb2-e271-4044-ab8e-cc5a6335af53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:54:16.255540 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:16.255508 5615 generic.go:356] "Generic (PLEG): container finished" podID="1ac55cb2-e271-4044-ab8e-cc5a6335af53" containerID="b2188484e61d2824f9553f15dcf77ef8c998b440f86e4f0b001b6b5e3e913518" exitCode=0 Nov 03 21:54:16.255734 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:16.255579 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" event={"ID":"1ac55cb2-e271-4044-ab8e-cc5a6335af53","Type":"ContainerDied","Data":"b2188484e61d2824f9553f15dcf77ef8c998b440f86e4f0b001b6b5e3e913518"} Nov 03 21:54:16.256031 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:16.256016 5615 scope.go:122] "RemoveContainer" containerID="b2188484e61d2824f9553f15dcf77ef8c998b440f86e4f0b001b6b5e3e913518" Nov 03 21:54:16.257798 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:16.257771 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:54:16.257872 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:16.257854 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:54:16.257910 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:16.257897 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 21:54:17.275119 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:17.273894 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" event={"ID":"1ac55cb2-e271-4044-ab8e-cc5a6335af53","Type":"ContainerStarted","Data":"8dc8b17294eec07058e5b516f8669ffaa91ccfce26fd826141d88ad3f2d847db"} Nov 03 21:54:17.535473 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:17.535380 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:54:17.535694 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:17.535669 5615 patch_prober.go:29] interesting pod/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 03 21:54:17.535748 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:17.535719 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" podUID="1ac55cb2-e271-4044-ab8e-cc5a6335af53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 03 21:54:18.536555 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:18.536517 5615 patch_prober.go:29] interesting pod/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 03 21:54:18.536979 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:18.536574 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" podUID="1ac55cb2-e271-4044-ab8e-cc5a6335af53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 03 21:54:19.535663 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:19.535622 5615 patch_prober.go:29] interesting pod/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 03 21:54:19.535854 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:19.535694 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" podUID="1ac55cb2-e271-4044-ab8e-cc5a6335af53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 03 21:54:20.144317 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.144278 5615 patch_prober.go:29] interesting pod/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 03 21:54:20.144699 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.144346 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" podUID="0f2fd230-b1c2-4835-bc30-25437b149c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 03 21:54:20.268837 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.268810 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:54:20.294864 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.294829 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-istio-data\") pod \"0f2fd230-b1c2-4835-bc30-25437b149c02\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " Nov 03 21:54:20.295057 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.294940 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f2fd230-b1c2-4835-bc30-25437b149c02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f2fd230-b1c2-4835-bc30-25437b149c02-istiod-ca-cert\") pod \"0f2fd230-b1c2-4835-bc30-25437b149c02\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " Nov 03 21:54:20.295057 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.295005 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f2fd230-b1c2-4835-bc30-25437b149c02-istio-token\" (UniqueName: \"kubernetes.io/projected/0f2fd230-b1c2-4835-bc30-25437b149c02-istio-token\") pod \"0f2fd230-b1c2-4835-bc30-25437b149c02\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " Nov 03 21:54:20.295234 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.295060 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-credential-socket\") pod \"0f2fd230-b1c2-4835-bc30-25437b149c02\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " Nov 03 21:54:20.295234 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.295099 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f2fd230-b1c2-4835-bc30-25437b149c02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f2fd230-b1c2-4835-bc30-25437b149c02-istio-podinfo\") pod \"0f2fd230-b1c2-4835-bc30-25437b149c02\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " Nov 03 21:54:20.295234 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.295128 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f2fd230-b1c2-4835-bc30-25437b149c02-kube-api-access-zmzzq\" (UniqueName: \"kubernetes.io/projected/0f2fd230-b1c2-4835-bc30-25437b149c02-kube-api-access-zmzzq\") pod \"0f2fd230-b1c2-4835-bc30-25437b149c02\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " Nov 03 21:54:20.295234 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.295172 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-istio-data" pod "0f2fd230-b1c2-4835-bc30-25437b149c02" (UID: "0f2fd230-b1c2-4835-bc30-25437b149c02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:54:20.295234 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.295199 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-workload-certs\") pod \"0f2fd230-b1c2-4835-bc30-25437b149c02\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " Nov 03 21:54:20.295496 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.295284 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-workload-socket\") pod \"0f2fd230-b1c2-4835-bc30-25437b149c02\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " Nov 03 21:54:20.295496 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.295367 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-istio-envoy\") pod \"0f2fd230-b1c2-4835-bc30-25437b149c02\" (UID: \"0f2fd230-b1c2-4835-bc30-25437b149c02\") " Nov 03 21:54:20.295496 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.295437 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f2fd230-b1c2-4835-bc30-25437b149c02-istiod-ca-cert" pod "0f2fd230-b1c2-4835-bc30-25437b149c02" (UID: "0f2fd230-b1c2-4835-bc30-25437b149c02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:54:20.295611 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.295554 5615 generic.go:356] "Generic (PLEG): container finished" podID="0f2fd230-b1c2-4835-bc30-25437b149c02" containerID="d69b51f960d1deb1d8a5d2199d451990cf1d9a032ceede885b1af47b0709e139" exitCode=0 Nov 03 21:54:20.295762 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.295736 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" event={"ID":"0f2fd230-b1c2-4835-bc30-25437b149c02","Type":"ContainerDied","Data":"d69b51f960d1deb1d8a5d2199d451990cf1d9a032ceede885b1af47b0709e139"} Nov 03 21:54:20.295861 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.295796 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" event={"ID":"0f2fd230-b1c2-4835-bc30-25437b149c02","Type":"ContainerDied","Data":"2a37c2f3fb7fd45f549813f3b87b31241010fff21153e5793410c60caf291350"} Nov 03 21:54:20.295861 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.295828 5615 scope.go:122] "RemoveContainer" containerID="d69b51f960d1deb1d8a5d2199d451990cf1d9a032ceede885b1af47b0709e139" Nov 03 21:54:20.295987 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.295963 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:20.295987 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.295980 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f2fd230-b1c2-4835-bc30-25437b149c02-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:20.296139 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.296008 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z" Nov 03 21:54:20.296570 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.296533 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-credential-socket" pod "0f2fd230-b1c2-4835-bc30-25437b149c02" (UID: "0f2fd230-b1c2-4835-bc30-25437b149c02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:54:20.298257 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.298165 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-workload-socket" pod "0f2fd230-b1c2-4835-bc30-25437b149c02" (UID: "0f2fd230-b1c2-4835-bc30-25437b149c02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:54:20.298257 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.298232 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-workload-certs" pod "0f2fd230-b1c2-4835-bc30-25437b149c02" (UID: "0f2fd230-b1c2-4835-bc30-25437b149c02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:54:20.307590 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.307562 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f2fd230-b1c2-4835-bc30-25437b149c02-istio-podinfo" pod "0f2fd230-b1c2-4835-bc30-25437b149c02" (UID: "0f2fd230-b1c2-4835-bc30-25437b149c02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:54:20.307757 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.307612 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2fd230-b1c2-4835-bc30-25437b149c02-istio-token" pod "0f2fd230-b1c2-4835-bc30-25437b149c02" (UID: "0f2fd230-b1c2-4835-bc30-25437b149c02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:54:20.329295 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.329249 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-istio-envoy" pod "0f2fd230-b1c2-4835-bc30-25437b149c02" (UID: "0f2fd230-b1c2-4835-bc30-25437b149c02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:54:20.329805 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.329766 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2fd230-b1c2-4835-bc30-25437b149c02-kube-api-access-zmzzq" pod "0f2fd230-b1c2-4835-bc30-25437b149c02" (UID: "0f2fd230-b1c2-4835-bc30-25437b149c02"). InnerVolumeSpecName "kube-api-access-zmzzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:54:20.345322 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.345297 5615 scope.go:122] "RemoveContainer" containerID="858c66df7ca402d81c7079c5df4851c1c0b37b2f543db77f6cc21bdd9eef5cc7" Nov 03 21:54:20.354928 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.354906 5615 scope.go:122] "RemoveContainer" containerID="d69b51f960d1deb1d8a5d2199d451990cf1d9a032ceede885b1af47b0709e139" Nov 03 21:54:20.355233 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:54:20.355212 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d69b51f960d1deb1d8a5d2199d451990cf1d9a032ceede885b1af47b0709e139\": container with ID starting with d69b51f960d1deb1d8a5d2199d451990cf1d9a032ceede885b1af47b0709e139 not found: ID does not exist" containerID="d69b51f960d1deb1d8a5d2199d451990cf1d9a032ceede885b1af47b0709e139" Nov 03 21:54:20.355299 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.355243 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d69b51f960d1deb1d8a5d2199d451990cf1d9a032ceede885b1af47b0709e139"} err="failed to get container status \"d69b51f960d1deb1d8a5d2199d451990cf1d9a032ceede885b1af47b0709e139\": rpc error: code = NotFound desc = could not find container \"d69b51f960d1deb1d8a5d2199d451990cf1d9a032ceede885b1af47b0709e139\": container with ID starting with d69b51f960d1deb1d8a5d2199d451990cf1d9a032ceede885b1af47b0709e139 not found: ID does not exist" Nov 03 21:54:20.355299 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.355262 5615 scope.go:122] "RemoveContainer" containerID="858c66df7ca402d81c7079c5df4851c1c0b37b2f543db77f6cc21bdd9eef5cc7" Nov 03 21:54:20.355509 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:54:20.355490 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"858c66df7ca402d81c7079c5df4851c1c0b37b2f543db77f6cc21bdd9eef5cc7\": container with ID starting with 858c66df7ca402d81c7079c5df4851c1c0b37b2f543db77f6cc21bdd9eef5cc7 not found: ID does not exist" containerID="858c66df7ca402d81c7079c5df4851c1c0b37b2f543db77f6cc21bdd9eef5cc7" Nov 03 21:54:20.355550 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.355516 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"858c66df7ca402d81c7079c5df4851c1c0b37b2f543db77f6cc21bdd9eef5cc7"} err="failed to get container status \"858c66df7ca402d81c7079c5df4851c1c0b37b2f543db77f6cc21bdd9eef5cc7\": rpc error: code = NotFound desc = could not find container \"858c66df7ca402d81c7079c5df4851c1c0b37b2f543db77f6cc21bdd9eef5cc7\": container with ID starting with 858c66df7ca402d81c7079c5df4851c1c0b37b2f543db77f6cc21bdd9eef5cc7 not found: ID does not exist" Nov 03 21:54:20.398970 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.397803 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:20.398970 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.397835 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f2fd230-b1c2-4835-bc30-25437b149c02-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:20.398970 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.397851 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:20.398970 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.397864 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f2fd230-b1c2-4835-bc30-25437b149c02-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:20.398970 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.397877 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmzzq\" (UniqueName: \"kubernetes.io/projected/0f2fd230-b1c2-4835-bc30-25437b149c02-kube-api-access-zmzzq\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:20.398970 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.397890 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:20.398970 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.397903 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2fd230-b1c2-4835-bc30-25437b149c02-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:20.432100 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.429137 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-9e9142-65cfd77f76-x8thm" podUID="4b18bd0d-0a30-4471-8ced-27cce9e7b5b0" containerName="payload-processing" containerID="cri-o://3a1085efe9ef68d5e27e54a6e3e0e864744f9faea71624df6705bb0a3c9f9ec5" gracePeriod=25 Nov 03 21:54:20.535911 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.535835 5615 patch_prober.go:29] interesting pod/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 03 21:54:20.535911 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.535891 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" podUID="1ac55cb2-e271-4044-ab8e-cc5a6335af53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 03 21:54:20.543961 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.543924 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e9142-7f8b6dbf5f-5vq49" podUID="fb102e63-94c5-4eed-ad81-dbb253fbc44d" containerName="payload-pre-processing" containerID="cri-o://0e271f3643e8ccb87a91e77d250e2fdb018e38d06c2539f473131b52a5c9160d" gracePeriod=25 Nov 03 21:54:20.769817 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.769781 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z"] Nov 03 21:54:20.773339 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.773306 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-965fe8-openshift-default-bff8889bf-2xf4z"] Nov 03 21:54:20.778295 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.778269 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9e9142-65cfd77f76-x8thm" Nov 03 21:54:20.863520 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.863483 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b18bd0d-0a30-4471-8ced-27cce9e7b5b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b18bd0d-0a30-4471-8ced-27cce9e7b5b0-plugins-config-volume\") pod \"4b18bd0d-0a30-4471-8ced-27cce9e7b5b0\" (UID: \"4b18bd0d-0a30-4471-8ced-27cce9e7b5b0\") " Nov 03 21:54:20.864207 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.864178 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b18bd0d-0a30-4471-8ced-27cce9e7b5b0-kube-api-access-njgq9\" (UniqueName: \"kubernetes.io/projected/4b18bd0d-0a30-4471-8ced-27cce9e7b5b0-kube-api-access-njgq9\") pod \"4b18bd0d-0a30-4471-8ced-27cce9e7b5b0\" (UID: \"4b18bd0d-0a30-4471-8ced-27cce9e7b5b0\") " Nov 03 21:54:20.864887 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.863887 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b18bd0d-0a30-4471-8ced-27cce9e7b5b0-plugins-config-volume" pod "4b18bd0d-0a30-4471-8ced-27cce9e7b5b0" (UID: "4b18bd0d-0a30-4471-8ced-27cce9e7b5b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:54:20.867772 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.867750 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b18bd0d-0a30-4471-8ced-27cce9e7b5b0-kube-api-access-njgq9" pod "4b18bd0d-0a30-4471-8ced-27cce9e7b5b0" (UID: "4b18bd0d-0a30-4471-8ced-27cce9e7b5b0"). InnerVolumeSpecName "kube-api-access-njgq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:54:20.948435 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.948408 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e9142-7f8b6dbf5f-5vq49" Nov 03 21:54:20.965822 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.965785 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b18bd0d-0a30-4471-8ced-27cce9e7b5b0-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:20.965822 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:20.965817 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njgq9\" (UniqueName: \"kubernetes.io/projected/4b18bd0d-0a30-4471-8ced-27cce9e7b5b0-kube-api-access-njgq9\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:21.066552 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:21.066511 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb102e63-94c5-4eed-ad81-dbb253fbc44d-kube-api-access-fnc5w\" (UniqueName: \"kubernetes.io/projected/fb102e63-94c5-4eed-ad81-dbb253fbc44d-kube-api-access-fnc5w\") pod \"fb102e63-94c5-4eed-ad81-dbb253fbc44d\" (UID: \"fb102e63-94c5-4eed-ad81-dbb253fbc44d\") " Nov 03 21:54:21.066762 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:21.066628 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb102e63-94c5-4eed-ad81-dbb253fbc44d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb102e63-94c5-4eed-ad81-dbb253fbc44d-plugins-config-volume\") pod \"fb102e63-94c5-4eed-ad81-dbb253fbc44d\" (UID: \"fb102e63-94c5-4eed-ad81-dbb253fbc44d\") " Nov 03 21:54:21.067017 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:21.066995 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb102e63-94c5-4eed-ad81-dbb253fbc44d-plugins-config-volume" pod "fb102e63-94c5-4eed-ad81-dbb253fbc44d" (UID: "fb102e63-94c5-4eed-ad81-dbb253fbc44d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:54:21.067165 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:21.067147 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb102e63-94c5-4eed-ad81-dbb253fbc44d-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:21.068946 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:21.068925 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb102e63-94c5-4eed-ad81-dbb253fbc44d-kube-api-access-fnc5w" pod "fb102e63-94c5-4eed-ad81-dbb253fbc44d" (UID: "fb102e63-94c5-4eed-ad81-dbb253fbc44d"). InnerVolumeSpecName "kube-api-access-fnc5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:54:21.168961 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:21.168871 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnc5w\" (UniqueName: \"kubernetes.io/projected/fb102e63-94c5-4eed-ad81-dbb253fbc44d-kube-api-access-fnc5w\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:21.328312 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:21.328272 5615 generic.go:356] "Generic (PLEG): container finished" podID="fb102e63-94c5-4eed-ad81-dbb253fbc44d" containerID="0e271f3643e8ccb87a91e77d250e2fdb018e38d06c2539f473131b52a5c9160d" exitCode=0 Nov 03 21:54:21.328522 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:21.328319 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e9142-7f8b6dbf5f-5vq49" Nov 03 21:54:21.328522 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:21.328354 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e9142-7f8b6dbf5f-5vq49" event={"ID":"fb102e63-94c5-4eed-ad81-dbb253fbc44d","Type":"ContainerDied","Data":"0e271f3643e8ccb87a91e77d250e2fdb018e38d06c2539f473131b52a5c9160d"} Nov 03 21:54:21.328522 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:21.328409 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e9142-7f8b6dbf5f-5vq49" event={"ID":"fb102e63-94c5-4eed-ad81-dbb253fbc44d","Type":"ContainerDied","Data":"52360170ba37b29af53796441faf7e0645dc85048781003cd6247f885dadccd5"} Nov 03 21:54:21.328522 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:21.328443 5615 scope.go:122] "RemoveContainer" containerID="0e271f3643e8ccb87a91e77d250e2fdb018e38d06c2539f473131b52a5c9160d" Nov 03 21:54:21.332349 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:21.332315 5615 generic.go:356] "Generic (PLEG): container finished" podID="4b18bd0d-0a30-4471-8ced-27cce9e7b5b0" containerID="3a1085efe9ef68d5e27e54a6e3e0e864744f9faea71624df6705bb0a3c9f9ec5" exitCode=0 Nov 03 21:54:21.332503 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:21.332397 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9e9142-65cfd77f76-x8thm" event={"ID":"4b18bd0d-0a30-4471-8ced-27cce9e7b5b0","Type":"ContainerDied","Data":"3a1085efe9ef68d5e27e54a6e3e0e864744f9faea71624df6705bb0a3c9f9ec5"} Nov 03 21:54:21.332503 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:21.332459 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9e9142-65cfd77f76-x8thm" event={"ID":"4b18bd0d-0a30-4471-8ced-27cce9e7b5b0","Type":"ContainerDied","Data":"17ff6dccd4bca0816d9eb12b0e62c12ff453cfc53fb614bfe38ea3d606534f98"} Nov 03 21:54:21.332624 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:21.332514 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9e9142-65cfd77f76-x8thm" Nov 03 21:54:21.344771 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:21.344745 5615 scope.go:122] "RemoveContainer" containerID="0e271f3643e8ccb87a91e77d250e2fdb018e38d06c2539f473131b52a5c9160d" Nov 03 21:54:21.347502 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:54:21.345632 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e271f3643e8ccb87a91e77d250e2fdb018e38d06c2539f473131b52a5c9160d\": container with ID starting with 0e271f3643e8ccb87a91e77d250e2fdb018e38d06c2539f473131b52a5c9160d not found: ID does not exist" containerID="0e271f3643e8ccb87a91e77d250e2fdb018e38d06c2539f473131b52a5c9160d" Nov 03 21:54:21.347502 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:21.345673 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e271f3643e8ccb87a91e77d250e2fdb018e38d06c2539f473131b52a5c9160d"} err="failed to get container status \"0e271f3643e8ccb87a91e77d250e2fdb018e38d06c2539f473131b52a5c9160d\": rpc error: code = NotFound desc = could not find container \"0e271f3643e8ccb87a91e77d250e2fdb018e38d06c2539f473131b52a5c9160d\": container with ID starting with 0e271f3643e8ccb87a91e77d250e2fdb018e38d06c2539f473131b52a5c9160d not found: ID does not exist" Nov 03 21:54:21.347502 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:21.345699 5615 scope.go:122] "RemoveContainer" containerID="3a1085efe9ef68d5e27e54a6e3e0e864744f9faea71624df6705bb0a3c9f9ec5" Nov 03 21:54:21.356459 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:21.356434 5615 scope.go:122] "RemoveContainer" containerID="3a1085efe9ef68d5e27e54a6e3e0e864744f9faea71624df6705bb0a3c9f9ec5" Nov 03 21:54:21.356811 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:54:21.356785 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a1085efe9ef68d5e27e54a6e3e0e864744f9faea71624df6705bb0a3c9f9ec5\": container with ID starting with 3a1085efe9ef68d5e27e54a6e3e0e864744f9faea71624df6705bb0a3c9f9ec5 not found: ID does not exist" containerID="3a1085efe9ef68d5e27e54a6e3e0e864744f9faea71624df6705bb0a3c9f9ec5" Nov 03 21:54:21.356905 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:21.356817 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a1085efe9ef68d5e27e54a6e3e0e864744f9faea71624df6705bb0a3c9f9ec5"} err="failed to get container status \"3a1085efe9ef68d5e27e54a6e3e0e864744f9faea71624df6705bb0a3c9f9ec5\": rpc error: code = NotFound desc = could not find container \"3a1085efe9ef68d5e27e54a6e3e0e864744f9faea71624df6705bb0a3c9f9ec5\": container with ID starting with 3a1085efe9ef68d5e27e54a6e3e0e864744f9faea71624df6705bb0a3c9f9ec5 not found: ID does not exist" Nov 03 21:54:21.496195 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:21.496153 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9e9142-65cfd77f76-x8thm"] Nov 03 21:54:21.522761 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:21.522720 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9e9142-65cfd77f76-x8thm"] Nov 03 21:54:21.536276 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:21.536240 5615 patch_prober.go:29] interesting pod/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 03 21:54:21.536476 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:21.536307 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" podUID="1ac55cb2-e271-4044-ab8e-cc5a6335af53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 03 21:54:21.629987 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:21.629951 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9e9142-7f8b6dbf5f-5vq49"] Nov 03 21:54:21.633095 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:21.633054 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9e9142-7f8b6dbf5f-5vq49"] Nov 03 21:54:21.652930 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:21.652890 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f2fd230-b1c2-4835-bc30-25437b149c02" path="/var/lib/kubelet/pods/0f2fd230-b1c2-4835-bc30-25437b149c02/volumes" Nov 03 21:54:21.653468 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:21.653446 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b18bd0d-0a30-4471-8ced-27cce9e7b5b0" path="/var/lib/kubelet/pods/4b18bd0d-0a30-4471-8ced-27cce9e7b5b0/volumes" Nov 03 21:54:21.653937 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:21.653917 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb102e63-94c5-4eed-ad81-dbb253fbc44d" path="/var/lib/kubelet/pods/fb102e63-94c5-4eed-ad81-dbb253fbc44d/volumes" Nov 03 21:54:22.565819 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:22.565772 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:54:22.566824 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:22.566365 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:54:22.567971 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:22.567947 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:54:40.619033 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:54:40.618938 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:54:51.359491 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:51.359449 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5"] Nov 03 21:54:51.360158 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:51.359682 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" podUID="884c8308-cf0b-496d-9f55-bb6563fe0238" containerName="istio-proxy" containerID="cri-o://7d264a294afeecb09d04c751e3d95b241f79f5d3530f1e6860b948c2ddf9783c" gracePeriod=30 Nov 03 21:54:54.149844 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:54.149799 5615 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:54:54.150294 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:54.149868 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" podUID="884c8308-cf0b-496d-9f55-bb6563fe0238" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:54:56.629656 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.629574 5615 generic.go:356] "Generic (PLEG): container finished" podID="884c8308-cf0b-496d-9f55-bb6563fe0238" containerID="7d264a294afeecb09d04c751e3d95b241f79f5d3530f1e6860b948c2ddf9783c" exitCode=0 Nov 03 21:54:56.629656 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.629643 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" event={"ID":"884c8308-cf0b-496d-9f55-bb6563fe0238","Type":"ContainerDied","Data":"7d264a294afeecb09d04c751e3d95b241f79f5d3530f1e6860b948c2ddf9783c"} Nov 03 21:54:56.630185 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.629702 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" event={"ID":"884c8308-cf0b-496d-9f55-bb6563fe0238","Type":"ContainerDied","Data":"b426577046db478a746af3b9734d2c67daea67783af0ef2209b701d22c169cd6"} Nov 03 21:54:56.630185 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.629720 5615 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b426577046db478a746af3b9734d2c67daea67783af0ef2209b701d22c169cd6" Nov 03 21:54:56.630185 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.629733 5615 scope.go:122] "RemoveContainer" containerID="46727b5c84e14613cb6fd6dbdc300439f0b7bba72f858cbd7927e91592837e79" Nov 03 21:54:56.640931 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.640906 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:54:56.729436 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.729394 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884c8308-cf0b-496d-9f55-bb6563fe0238-istio-token\" (UniqueName: \"kubernetes.io/projected/884c8308-cf0b-496d-9f55-bb6563fe0238-istio-token\") pod \"884c8308-cf0b-496d-9f55-bb6563fe0238\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " Nov 03 21:54:56.729623 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.729498 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-istio-envoy\") pod \"884c8308-cf0b-496d-9f55-bb6563fe0238\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " Nov 03 21:54:56.729623 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.729585 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-workload-certs\") pod \"884c8308-cf0b-496d-9f55-bb6563fe0238\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " Nov 03 21:54:56.729734 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.729693 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884c8308-cf0b-496d-9f55-bb6563fe0238-kube-api-access-2c8xv\" (UniqueName: \"kubernetes.io/projected/884c8308-cf0b-496d-9f55-bb6563fe0238-kube-api-access-2c8xv\") pod \"884c8308-cf0b-496d-9f55-bb6563fe0238\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " Nov 03 21:54:56.729796 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.729744 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-workload-socket\") pod \"884c8308-cf0b-496d-9f55-bb6563fe0238\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " Nov 03 21:54:56.729796 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.729774 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/884c8308-cf0b-496d-9f55-bb6563fe0238-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/884c8308-cf0b-496d-9f55-bb6563fe0238-istiod-ca-cert\") pod \"884c8308-cf0b-496d-9f55-bb6563fe0238\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " Nov 03 21:54:56.729908 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.729823 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/884c8308-cf0b-496d-9f55-bb6563fe0238-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/884c8308-cf0b-496d-9f55-bb6563fe0238-istio-podinfo\") pod \"884c8308-cf0b-496d-9f55-bb6563fe0238\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " Nov 03 21:54:56.729908 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.729865 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-istio-data\") pod \"884c8308-cf0b-496d-9f55-bb6563fe0238\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " Nov 03 21:54:56.730017 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.729911 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-workload-certs" pod "884c8308-cf0b-496d-9f55-bb6563fe0238" (UID: "884c8308-cf0b-496d-9f55-bb6563fe0238"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:54:56.730017 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.729974 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-credential-socket\") pod \"884c8308-cf0b-496d-9f55-bb6563fe0238\" (UID: \"884c8308-cf0b-496d-9f55-bb6563fe0238\") " Nov 03 21:54:56.730148 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.730119 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-istio-data" pod "884c8308-cf0b-496d-9f55-bb6563fe0238" (UID: "884c8308-cf0b-496d-9f55-bb6563fe0238"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:54:56.730237 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.730208 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/884c8308-cf0b-496d-9f55-bb6563fe0238-istiod-ca-cert" pod "884c8308-cf0b-496d-9f55-bb6563fe0238" (UID: "884c8308-cf0b-496d-9f55-bb6563fe0238"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:54:56.730339 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.730306 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-workload-socket" pod "884c8308-cf0b-496d-9f55-bb6563fe0238" (UID: "884c8308-cf0b-496d-9f55-bb6563fe0238"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:54:56.730432 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.730411 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-credential-socket" pod "884c8308-cf0b-496d-9f55-bb6563fe0238" (UID: "884c8308-cf0b-496d-9f55-bb6563fe0238"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:54:56.730784 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.730757 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:56.730852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.730785 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:56.730852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.730800 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/884c8308-cf0b-496d-9f55-bb6563fe0238-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:56.730852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.730816 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:56.730852 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.730829 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:56.731886 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.731862 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-istio-envoy" pod "884c8308-cf0b-496d-9f55-bb6563fe0238" (UID: "884c8308-cf0b-496d-9f55-bb6563fe0238"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:54:56.732222 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.732197 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/884c8308-cf0b-496d-9f55-bb6563fe0238-istio-podinfo" pod "884c8308-cf0b-496d-9f55-bb6563fe0238" (UID: "884c8308-cf0b-496d-9f55-bb6563fe0238"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:54:56.732327 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.732225 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884c8308-cf0b-496d-9f55-bb6563fe0238-istio-token" pod "884c8308-cf0b-496d-9f55-bb6563fe0238" (UID: "884c8308-cf0b-496d-9f55-bb6563fe0238"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:54:56.732327 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.732229 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884c8308-cf0b-496d-9f55-bb6563fe0238-kube-api-access-2c8xv" pod "884c8308-cf0b-496d-9f55-bb6563fe0238" (UID: "884c8308-cf0b-496d-9f55-bb6563fe0238"). InnerVolumeSpecName "kube-api-access-2c8xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:54:56.840580 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.836115 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/884c8308-cf0b-496d-9f55-bb6563fe0238-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:56.840580 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.836160 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/884c8308-cf0b-496d-9f55-bb6563fe0238-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:56.840580 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.836177 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/884c8308-cf0b-496d-9f55-bb6563fe0238-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:56.840580 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:56.836191 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c8xv\" (UniqueName: \"kubernetes.io/projected/884c8308-cf0b-496d-9f55-bb6563fe0238-kube-api-access-2c8xv\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:57.637959 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:57.637926 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5" Nov 03 21:54:57.927479 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:57.927391 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5"] Nov 03 21:54:57.931630 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:57.931599 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-76tc5"] Nov 03 21:54:59.653414 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:54:59.653377 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="884c8308-cf0b-496d-9f55-bb6563fe0238" path="/var/lib/kubelet/pods/884c8308-cf0b-496d-9f55-bb6563fe0238/volumes" Nov 03 21:55:10.619415 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:55:10.619370 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:55:27.328662 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:55:27.328618 5615 scope.go:122] "RemoveContainer" containerID="7d264a294afeecb09d04c751e3d95b241f79f5d3530f1e6860b948c2ddf9783c" Nov 03 21:55:38.193928 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:55:38.193889 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh"] Nov 03 21:55:38.194431 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:55:38.194119 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh" podUID="fbc63d42-9000-4c29-b2ff-2c4ef820b337" containerName="maas-api" containerID="cri-o://1bf865777583c3c9102a0e7b6f64798f27200442f76aa7642790cdc4c9e7ffc3" gracePeriod=30 Nov 03 21:55:38.498360 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:55:38.498326 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh" Nov 03 21:55:38.539436 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:55:38.539397 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbc63d42-9000-4c29-b2ff-2c4ef820b337-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbc63d42-9000-4c29-b2ff-2c4ef820b337-maas-api-tls\") pod \"fbc63d42-9000-4c29-b2ff-2c4ef820b337\" (UID: \"fbc63d42-9000-4c29-b2ff-2c4ef820b337\") " Nov 03 21:55:38.539612 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:55:38.539496 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbc63d42-9000-4c29-b2ff-2c4ef820b337-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc63d42-9000-4c29-b2ff-2c4ef820b337-metrics-certs\") pod \"fbc63d42-9000-4c29-b2ff-2c4ef820b337\" (UID: \"fbc63d42-9000-4c29-b2ff-2c4ef820b337\") " Nov 03 21:55:38.539612 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:55:38.539558 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbc63d42-9000-4c29-b2ff-2c4ef820b337-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbc63d42-9000-4c29-b2ff-2c4ef820b337-openshift-service-ca\") pod \"fbc63d42-9000-4c29-b2ff-2c4ef820b337\" (UID: \"fbc63d42-9000-4c29-b2ff-2c4ef820b337\") " Nov 03 21:55:38.539706 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:55:38.539679 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbc63d42-9000-4c29-b2ff-2c4ef820b337-kube-api-access-k5bb4\" (UniqueName: \"kubernetes.io/projected/fbc63d42-9000-4c29-b2ff-2c4ef820b337-kube-api-access-k5bb4\") pod \"fbc63d42-9000-4c29-b2ff-2c4ef820b337\" (UID: \"fbc63d42-9000-4c29-b2ff-2c4ef820b337\") " Nov 03 21:55:38.539933 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:55:38.539908 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbc63d42-9000-4c29-b2ff-2c4ef820b337-openshift-service-ca" pod "fbc63d42-9000-4c29-b2ff-2c4ef820b337" (UID: "fbc63d42-9000-4c29-b2ff-2c4ef820b337"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:55:38.540297 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:55:38.540276 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbc63d42-9000-4c29-b2ff-2c4ef820b337-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:55:38.541772 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:55:38.541743 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbc63d42-9000-4c29-b2ff-2c4ef820b337-maas-api-tls" pod "fbc63d42-9000-4c29-b2ff-2c4ef820b337" (UID: "fbc63d42-9000-4c29-b2ff-2c4ef820b337"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:55:38.541889 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:55:38.541781 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbc63d42-9000-4c29-b2ff-2c4ef820b337-metrics-certs" pod "fbc63d42-9000-4c29-b2ff-2c4ef820b337" (UID: "fbc63d42-9000-4c29-b2ff-2c4ef820b337"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:55:38.541889 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:55:38.541799 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbc63d42-9000-4c29-b2ff-2c4ef820b337-kube-api-access-k5bb4" pod "fbc63d42-9000-4c29-b2ff-2c4ef820b337" (UID: "fbc63d42-9000-4c29-b2ff-2c4ef820b337"). InnerVolumeSpecName "kube-api-access-k5bb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:55:38.641902 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:55:38.641861 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5bb4\" (UniqueName: \"kubernetes.io/projected/fbc63d42-9000-4c29-b2ff-2c4ef820b337-kube-api-access-k5bb4\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:55:38.641902 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:55:38.641900 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbc63d42-9000-4c29-b2ff-2c4ef820b337-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:55:38.642130 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:55:38.641913 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc63d42-9000-4c29-b2ff-2c4ef820b337-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:55:38.915063 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:55:38.915027 5615 generic.go:356] "Generic (PLEG): container finished" podID="fbc63d42-9000-4c29-b2ff-2c4ef820b337" containerID="1bf865777583c3c9102a0e7b6f64798f27200442f76aa7642790cdc4c9e7ffc3" exitCode=0 Nov 03 21:55:38.915308 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:55:38.915109 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh" Nov 03 21:55:38.915308 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:55:38.915122 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh" event={"ID":"fbc63d42-9000-4c29-b2ff-2c4ef820b337","Type":"ContainerDied","Data":"1bf865777583c3c9102a0e7b6f64798f27200442f76aa7642790cdc4c9e7ffc3"} Nov 03 21:55:38.915308 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:55:38.915188 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh" event={"ID":"fbc63d42-9000-4c29-b2ff-2c4ef820b337","Type":"ContainerDied","Data":"fa6d70d7276c5df622f644422b6965bf33a8b33efcee62f58ad536c676982b19"} Nov 03 21:55:38.915308 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:55:38.915211 5615 scope.go:122] "RemoveContainer" containerID="1bf865777583c3c9102a0e7b6f64798f27200442f76aa7642790cdc4c9e7ffc3" Nov 03 21:55:38.926640 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:55:38.926618 5615 scope.go:122] "RemoveContainer" containerID="1bf865777583c3c9102a0e7b6f64798f27200442f76aa7642790cdc4c9e7ffc3" Nov 03 21:55:38.926928 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:55:38.926906 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bf865777583c3c9102a0e7b6f64798f27200442f76aa7642790cdc4c9e7ffc3\": container with ID starting with 1bf865777583c3c9102a0e7b6f64798f27200442f76aa7642790cdc4c9e7ffc3 not found: ID does not exist" containerID="1bf865777583c3c9102a0e7b6f64798f27200442f76aa7642790cdc4c9e7ffc3" Nov 03 21:55:38.926988 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:55:38.926937 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bf865777583c3c9102a0e7b6f64798f27200442f76aa7642790cdc4c9e7ffc3"} err="failed to get container status \"1bf865777583c3c9102a0e7b6f64798f27200442f76aa7642790cdc4c9e7ffc3\": rpc error: code = NotFound desc = could not find container \"1bf865777583c3c9102a0e7b6f64798f27200442f76aa7642790cdc4c9e7ffc3\": container with ID starting with 1bf865777583c3c9102a0e7b6f64798f27200442f76aa7642790cdc4c9e7ffc3 not found: ID does not exist" Nov 03 21:55:39.108235 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:55:39.108165 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh"] Nov 03 21:55:39.122671 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:55:39.122621 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9e9142-6bfd7dbfbd-gjrlh"] Nov 03 21:55:39.653205 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:55:39.653168 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbc63d42-9000-4c29-b2ff-2c4ef820b337" path="/var/lib/kubelet/pods/fbc63d42-9000-4c29-b2ff-2c4ef820b337/volumes" Nov 03 21:55:40.620295 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:55:40.620252 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:56:03.675595 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:03.675559 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v"] Nov 03 21:56:03.676128 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:03.675757 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" podUID="38287625-1740-41e8-937c-a7e55e2d8c88" containerName="istio-proxy" containerID="cri-o://12107666eae12d0141514ca5553b938284188c16b93fc20d97fe15e682f8cdb7" gracePeriod=30 Nov 03 21:56:09.049507 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.049479 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:56:09.115036 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.114999 5615 generic.go:356] "Generic (PLEG): container finished" podID="38287625-1740-41e8-937c-a7e55e2d8c88" containerID="12107666eae12d0141514ca5553b938284188c16b93fc20d97fe15e682f8cdb7" exitCode=0 Nov 03 21:56:09.115235 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.115050 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" Nov 03 21:56:09.115235 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.115097 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" event={"ID":"38287625-1740-41e8-937c-a7e55e2d8c88","Type":"ContainerDied","Data":"12107666eae12d0141514ca5553b938284188c16b93fc20d97fe15e682f8cdb7"} Nov 03 21:56:09.115235 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.115160 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v" event={"ID":"38287625-1740-41e8-937c-a7e55e2d8c88","Type":"ContainerDied","Data":"82f0f5b01993778cce9d344deb61c0a2eb9fa5e4bdc42015fd8f26712db9bd64"} Nov 03 21:56:09.115235 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.115184 5615 scope.go:122] "RemoveContainer" containerID="12107666eae12d0141514ca5553b938284188c16b93fc20d97fe15e682f8cdb7" Nov 03 21:56:09.125794 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.125773 5615 scope.go:122] "RemoveContainer" containerID="66fb649ee98dd0caf029761a3bd756229e952475fe89f672b7c105ff83cfae24" Nov 03 21:56:09.135253 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.135231 5615 scope.go:122] "RemoveContainer" containerID="12107666eae12d0141514ca5553b938284188c16b93fc20d97fe15e682f8cdb7" Nov 03 21:56:09.135586 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:56:09.135560 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12107666eae12d0141514ca5553b938284188c16b93fc20d97fe15e682f8cdb7\": container with ID starting with 12107666eae12d0141514ca5553b938284188c16b93fc20d97fe15e682f8cdb7 not found: ID does not exist" containerID="12107666eae12d0141514ca5553b938284188c16b93fc20d97fe15e682f8cdb7" Nov 03 21:56:09.135686 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.135591 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12107666eae12d0141514ca5553b938284188c16b93fc20d97fe15e682f8cdb7"} err="failed to get container status \"12107666eae12d0141514ca5553b938284188c16b93fc20d97fe15e682f8cdb7\": rpc error: code = NotFound desc = could not find container \"12107666eae12d0141514ca5553b938284188c16b93fc20d97fe15e682f8cdb7\": container with ID starting with 12107666eae12d0141514ca5553b938284188c16b93fc20d97fe15e682f8cdb7 not found: ID does not exist" Nov 03 21:56:09.135686 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.135609 5615 scope.go:122] "RemoveContainer" containerID="66fb649ee98dd0caf029761a3bd756229e952475fe89f672b7c105ff83cfae24" Nov 03 21:56:09.135860 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:56:09.135842 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66fb649ee98dd0caf029761a3bd756229e952475fe89f672b7c105ff83cfae24\": container with ID starting with 66fb649ee98dd0caf029761a3bd756229e952475fe89f672b7c105ff83cfae24 not found: ID does not exist" containerID="66fb649ee98dd0caf029761a3bd756229e952475fe89f672b7c105ff83cfae24" Nov 03 21:56:09.135904 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.135867 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66fb649ee98dd0caf029761a3bd756229e952475fe89f672b7c105ff83cfae24"} err="failed to get container status \"66fb649ee98dd0caf029761a3bd756229e952475fe89f672b7c105ff83cfae24\": rpc error: code = NotFound desc = could not find container \"66fb649ee98dd0caf029761a3bd756229e952475fe89f672b7c105ff83cfae24\": container with ID starting with 66fb649ee98dd0caf029761a3bd756229e952475fe89f672b7c105ff83cfae24 not found: ID does not exist" Nov 03 21:56:09.206092 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.206047 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-credential-socket\") pod \"38287625-1740-41e8-937c-a7e55e2d8c88\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " Nov 03 21:56:09.206270 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.206112 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38287625-1740-41e8-937c-a7e55e2d8c88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38287625-1740-41e8-937c-a7e55e2d8c88-istio-podinfo\") pod \"38287625-1740-41e8-937c-a7e55e2d8c88\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " Nov 03 21:56:09.206270 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.206142 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-istio-envoy\") pod \"38287625-1740-41e8-937c-a7e55e2d8c88\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " Nov 03 21:56:09.206270 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.206218 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-istio-data\") pod \"38287625-1740-41e8-937c-a7e55e2d8c88\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " Nov 03 21:56:09.206429 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.206298 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38287625-1740-41e8-937c-a7e55e2d8c88-istio-token\" (UniqueName: \"kubernetes.io/projected/38287625-1740-41e8-937c-a7e55e2d8c88-istio-token\") pod \"38287625-1740-41e8-937c-a7e55e2d8c88\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " Nov 03 21:56:09.206429 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.206320 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-credential-socket" pod "38287625-1740-41e8-937c-a7e55e2d8c88" (UID: "38287625-1740-41e8-937c-a7e55e2d8c88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:56:09.206429 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.206338 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38287625-1740-41e8-937c-a7e55e2d8c88-kube-api-access-fvwpj\" (UniqueName: \"kubernetes.io/projected/38287625-1740-41e8-937c-a7e55e2d8c88-kube-api-access-fvwpj\") pod \"38287625-1740-41e8-937c-a7e55e2d8c88\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " Nov 03 21:56:09.206429 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.206385 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-workload-socket\") pod \"38287625-1740-41e8-937c-a7e55e2d8c88\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " Nov 03 21:56:09.206429 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.206412 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38287625-1740-41e8-937c-a7e55e2d8c88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38287625-1740-41e8-937c-a7e55e2d8c88-istiod-ca-cert\") pod \"38287625-1740-41e8-937c-a7e55e2d8c88\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " Nov 03 21:56:09.206693 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.206462 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-workload-certs\") pod \"38287625-1740-41e8-937c-a7e55e2d8c88\" (UID: \"38287625-1740-41e8-937c-a7e55e2d8c88\") " Nov 03 21:56:09.206693 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.206622 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-istio-data" pod "38287625-1740-41e8-937c-a7e55e2d8c88" (UID: "38287625-1740-41e8-937c-a7e55e2d8c88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:56:09.206794 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.206693 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-workload-socket" pod "38287625-1740-41e8-937c-a7e55e2d8c88" (UID: "38287625-1740-41e8-937c-a7e55e2d8c88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:56:09.206899 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.206881 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-workload-certs" pod "38287625-1740-41e8-937c-a7e55e2d8c88" (UID: "38287625-1740-41e8-937c-a7e55e2d8c88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:56:09.206970 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.206826 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38287625-1740-41e8-937c-a7e55e2d8c88-istiod-ca-cert" pod "38287625-1740-41e8-937c-a7e55e2d8c88" (UID: "38287625-1740-41e8-937c-a7e55e2d8c88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:56:09.207186 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.207161 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38287625-1740-41e8-937c-a7e55e2d8c88-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:56:09.207282 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.207190 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:56:09.207282 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.207205 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:56:09.207282 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.207221 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:56:09.207282 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.207237 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:56:09.208524 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.208495 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38287625-1740-41e8-937c-a7e55e2d8c88-istio-podinfo" pod "38287625-1740-41e8-937c-a7e55e2d8c88" (UID: "38287625-1740-41e8-937c-a7e55e2d8c88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:56:09.208636 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.208553 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38287625-1740-41e8-937c-a7e55e2d8c88-istio-token" pod "38287625-1740-41e8-937c-a7e55e2d8c88" (UID: "38287625-1740-41e8-937c-a7e55e2d8c88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:56:09.208687 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.208657 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38287625-1740-41e8-937c-a7e55e2d8c88-kube-api-access-fvwpj" pod "38287625-1740-41e8-937c-a7e55e2d8c88" (UID: "38287625-1740-41e8-937c-a7e55e2d8c88"). InnerVolumeSpecName "kube-api-access-fvwpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:56:09.208687 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.208675 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-istio-envoy" pod "38287625-1740-41e8-937c-a7e55e2d8c88" (UID: "38287625-1740-41e8-937c-a7e55e2d8c88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:56:09.313961 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.313185 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38287625-1740-41e8-937c-a7e55e2d8c88-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:56:09.313961 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.313228 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvwpj\" (UniqueName: \"kubernetes.io/projected/38287625-1740-41e8-937c-a7e55e2d8c88-kube-api-access-fvwpj\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:56:09.313961 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.313245 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38287625-1740-41e8-937c-a7e55e2d8c88-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:56:09.313961 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.313258 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38287625-1740-41e8-937c-a7e55e2d8c88-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:56:09.458881 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:56:09.458845 5615 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38287625_1740_41e8_937c_a7e55e2d8c88.slice/crio-82f0f5b01993778cce9d344deb61c0a2eb9fa5e4bdc42015fd8f26712db9bd64\": RecentStats: unable to find data in memory cache]" Nov 03 21:56:09.539533 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.539499 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v"] Nov 03 21:56:09.543203 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.543156 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-76eaf8-openshift-default-56878fbf77-vrk8v"] Nov 03 21:56:09.652741 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:56:09.652662 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38287625-1740-41e8-937c-a7e55e2d8c88" path="/var/lib/kubelet/pods/38287625-1740-41e8-937c-a7e55e2d8c88/volumes" Nov 03 21:56:10.619901 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:56:10.619863 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:56:40.622556 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:56:40.622511 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:57:10.619018 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:57:10.618976 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:57:31.125087 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:31.125033 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7"] Nov 03 21:57:31.125553 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:31.125298 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" podUID="1ac55cb2-e271-4044-ab8e-cc5a6335af53" containerName="istio-proxy" containerID="cri-o://8dc8b17294eec07058e5b516f8669ffaa91ccfce26fd826141d88ad3f2d847db" gracePeriod=30 Nov 03 21:57:36.402229 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.402122 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:57:36.543170 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.543126 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-credential-socket\") pod \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " Nov 03 21:57:36.543347 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.543193 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istio-podinfo\") pod \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " Nov 03 21:57:36.543347 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.543244 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istio-token\" (UniqueName: \"kubernetes.io/projected/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istio-token\") pod \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " Nov 03 21:57:36.543429 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.543401 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-credential-socket" pod "1ac55cb2-e271-4044-ab8e-cc5a6335af53" (UID: "1ac55cb2-e271-4044-ab8e-cc5a6335af53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:57:36.543469 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.543439 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-workload-certs\") pod \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " Nov 03 21:57:36.543530 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.543518 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istio-data\") pod \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " Nov 03 21:57:36.543582 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.543557 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-workload-socket\") pod \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " Nov 03 21:57:36.543646 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.543607 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istiod-ca-cert\") pod \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " Nov 03 21:57:36.543706 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.543641 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-workload-certs" pod "1ac55cb2-e271-4044-ab8e-cc5a6335af53" (UID: "1ac55cb2-e271-4044-ab8e-cc5a6335af53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:57:36.543706 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.543689 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ac55cb2-e271-4044-ab8e-cc5a6335af53-kube-api-access-45v5t\" (UniqueName: \"kubernetes.io/projected/1ac55cb2-e271-4044-ab8e-cc5a6335af53-kube-api-access-45v5t\") pod \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " Nov 03 21:57:36.543820 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.543746 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istio-envoy\") pod \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\" (UID: \"1ac55cb2-e271-4044-ab8e-cc5a6335af53\") " Nov 03 21:57:36.543820 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.543766 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istio-data" pod "1ac55cb2-e271-4044-ab8e-cc5a6335af53" (UID: "1ac55cb2-e271-4044-ab8e-cc5a6335af53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:57:36.543820 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.543800 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-workload-socket" pod "1ac55cb2-e271-4044-ab8e-cc5a6335af53" (UID: "1ac55cb2-e271-4044-ab8e-cc5a6335af53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:57:36.544018 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.543995 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istiod-ca-cert" pod "1ac55cb2-e271-4044-ab8e-cc5a6335af53" (UID: "1ac55cb2-e271-4044-ab8e-cc5a6335af53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:57:36.544549 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.544520 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:57:36.544685 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.544551 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:57:36.544685 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.544565 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:57:36.544685 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.544578 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:57:36.544685 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.544590 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:57:36.545734 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.545705 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istio-podinfo" pod "1ac55cb2-e271-4044-ab8e-cc5a6335af53" (UID: "1ac55cb2-e271-4044-ab8e-cc5a6335af53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:57:36.545855 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.545774 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istio-token" pod "1ac55cb2-e271-4044-ab8e-cc5a6335af53" (UID: "1ac55cb2-e271-4044-ab8e-cc5a6335af53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:57:36.545855 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.545810 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ac55cb2-e271-4044-ab8e-cc5a6335af53-kube-api-access-45v5t" pod "1ac55cb2-e271-4044-ab8e-cc5a6335af53" (UID: "1ac55cb2-e271-4044-ab8e-cc5a6335af53"). InnerVolumeSpecName "kube-api-access-45v5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:57:36.545942 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.545921 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istio-envoy" pod "1ac55cb2-e271-4044-ab8e-cc5a6335af53" (UID: "1ac55cb2-e271-4044-ab8e-cc5a6335af53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:57:36.645492 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.645456 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45v5t\" (UniqueName: \"kubernetes.io/projected/1ac55cb2-e271-4044-ab8e-cc5a6335af53-kube-api-access-45v5t\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:57:36.645492 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.645488 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:57:36.645492 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.645498 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:57:36.645727 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.645508 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ac55cb2-e271-4044-ab8e-cc5a6335af53-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:57:36.709165 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.709046 5615 generic.go:356] "Generic (PLEG): container finished" podID="1ac55cb2-e271-4044-ab8e-cc5a6335af53" containerID="8dc8b17294eec07058e5b516f8669ffaa91ccfce26fd826141d88ad3f2d847db" exitCode=0 Nov 03 21:57:36.709165 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.709119 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" Nov 03 21:57:36.709165 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.709144 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" event={"ID":"1ac55cb2-e271-4044-ab8e-cc5a6335af53","Type":"ContainerDied","Data":"8dc8b17294eec07058e5b516f8669ffaa91ccfce26fd826141d88ad3f2d847db"} Nov 03 21:57:36.709428 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.709203 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7" event={"ID":"1ac55cb2-e271-4044-ab8e-cc5a6335af53","Type":"ContainerDied","Data":"3bf02e1c4a42ba73d8a8a19b60db7721b5d4deac7bbede55eb01c2997717b3c7"} Nov 03 21:57:36.709428 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.709228 5615 scope.go:122] "RemoveContainer" containerID="8dc8b17294eec07058e5b516f8669ffaa91ccfce26fd826141d88ad3f2d847db" Nov 03 21:57:36.720142 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.720069 5615 scope.go:122] "RemoveContainer" containerID="b2188484e61d2824f9553f15dcf77ef8c998b440f86e4f0b001b6b5e3e913518" Nov 03 21:57:36.729643 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.729617 5615 scope.go:122] "RemoveContainer" containerID="8dc8b17294eec07058e5b516f8669ffaa91ccfce26fd826141d88ad3f2d847db" Nov 03 21:57:36.730003 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:57:36.729981 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dc8b17294eec07058e5b516f8669ffaa91ccfce26fd826141d88ad3f2d847db\": container with ID starting with 8dc8b17294eec07058e5b516f8669ffaa91ccfce26fd826141d88ad3f2d847db not found: ID does not exist" containerID="8dc8b17294eec07058e5b516f8669ffaa91ccfce26fd826141d88ad3f2d847db" Nov 03 21:57:36.730121 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.730015 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc8b17294eec07058e5b516f8669ffaa91ccfce26fd826141d88ad3f2d847db"} err="failed to get container status \"8dc8b17294eec07058e5b516f8669ffaa91ccfce26fd826141d88ad3f2d847db\": rpc error: code = NotFound desc = could not find container \"8dc8b17294eec07058e5b516f8669ffaa91ccfce26fd826141d88ad3f2d847db\": container with ID starting with 8dc8b17294eec07058e5b516f8669ffaa91ccfce26fd826141d88ad3f2d847db not found: ID does not exist" Nov 03 21:57:36.730121 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.730037 5615 scope.go:122] "RemoveContainer" containerID="b2188484e61d2824f9553f15dcf77ef8c998b440f86e4f0b001b6b5e3e913518" Nov 03 21:57:36.730301 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:57:36.730278 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2188484e61d2824f9553f15dcf77ef8c998b440f86e4f0b001b6b5e3e913518\": container with ID starting with b2188484e61d2824f9553f15dcf77ef8c998b440f86e4f0b001b6b5e3e913518 not found: ID does not exist" containerID="b2188484e61d2824f9553f15dcf77ef8c998b440f86e4f0b001b6b5e3e913518" Nov 03 21:57:36.730349 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.730306 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2188484e61d2824f9553f15dcf77ef8c998b440f86e4f0b001b6b5e3e913518"} err="failed to get container status \"b2188484e61d2824f9553f15dcf77ef8c998b440f86e4f0b001b6b5e3e913518\": rpc error: code = NotFound desc = could not find container \"b2188484e61d2824f9553f15dcf77ef8c998b440f86e4f0b001b6b5e3e913518\": container with ID starting with b2188484e61d2824f9553f15dcf77ef8c998b440f86e4f0b001b6b5e3e913518 not found: ID does not exist" Nov 03 21:57:36.878626 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.878593 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7"] Nov 03 21:57:36.884847 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:36.884817 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0a0da1-openshift-default-7c4fdf4c8c-f8hr7"] Nov 03 21:57:37.658433 ip-10-0-129-192 kubenswrapper[5615]: I1103 21:57:37.658385 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ac55cb2-e271-4044-ab8e-cc5a6335af53" path="/var/lib/kubelet/pods/1ac55cb2-e271-4044-ab8e-cc5a6335af53/volumes" Nov 03 21:57:40.619449 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:57:40.619409 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:58:10.944326 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:58:10.944279 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:58:40.620486 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:58:40.620442 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:59:10.663594 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:59:10.663552 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 21:59:40.619556 ip-10-0-129-192 kubenswrapper[5615]: E1103 21:59:40.619497 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:00:08.236483 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.236442 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx"] Nov 03 22:00:08.237046 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.237031 5615 cpu_manager.go:414] "containerMap: removing container" podUID="4b18bd0d-0a30-4471-8ced-27cce9e7b5b0" containerName="payload-processing" Nov 03 22:00:08.237135 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.237046 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b18bd0d-0a30-4471-8ced-27cce9e7b5b0" containerName="payload-processing" Nov 03 22:00:08.237135 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.237070 5615 cpu_manager.go:414] "containerMap: removing container" podUID="884c8308-cf0b-496d-9f55-bb6563fe0238" containerName="istio-proxy" Nov 03 22:00:08.237135 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.237088 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884c8308-cf0b-496d-9f55-bb6563fe0238" containerName="istio-proxy" Nov 03 22:00:08.237135 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.237098 5615 cpu_manager.go:414] "containerMap: removing container" podUID="38287625-1740-41e8-937c-a7e55e2d8c88" containerName="istio-proxy" Nov 03 22:00:08.237135 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.237104 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38287625-1740-41e8-937c-a7e55e2d8c88" containerName="istio-proxy" Nov 03 22:00:08.237135 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.237119 5615 cpu_manager.go:414] "containerMap: removing container" podUID="1ac55cb2-e271-4044-ab8e-cc5a6335af53" containerName="istio-proxy" Nov 03 22:00:08.237135 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.237126 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ac55cb2-e271-4044-ab8e-cc5a6335af53" containerName="istio-proxy" Nov 03 22:00:08.237135 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.237134 5615 cpu_manager.go:414] "containerMap: removing container" podUID="0f2fd230-b1c2-4835-bc30-25437b149c02" containerName="istio-proxy" Nov 03 22:00:08.237135 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.237138 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f2fd230-b1c2-4835-bc30-25437b149c02" containerName="istio-proxy" Nov 03 22:00:08.237476 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.237152 5615 cpu_manager.go:414] "containerMap: removing container" podUID="0f2fd230-b1c2-4835-bc30-25437b149c02" containerName="istio-proxy" Nov 03 22:00:08.237476 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.237156 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f2fd230-b1c2-4835-bc30-25437b149c02" containerName="istio-proxy" Nov 03 22:00:08.237476 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.237170 5615 cpu_manager.go:414] "containerMap: removing container" podUID="fb102e63-94c5-4eed-ad81-dbb253fbc44d" containerName="payload-pre-processing" Nov 03 22:00:08.237476 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.237174 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb102e63-94c5-4eed-ad81-dbb253fbc44d" containerName="payload-pre-processing" Nov 03 22:00:08.237476 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.237187 5615 cpu_manager.go:414] "containerMap: removing container" podUID="1ac55cb2-e271-4044-ab8e-cc5a6335af53" containerName="istio-proxy" Nov 03 22:00:08.237476 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.237192 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ac55cb2-e271-4044-ab8e-cc5a6335af53" containerName="istio-proxy" Nov 03 22:00:08.237476 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.237200 5615 cpu_manager.go:414] "containerMap: removing container" podUID="fbc63d42-9000-4c29-b2ff-2c4ef820b337" containerName="maas-api" Nov 03 22:00:08.237476 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.237204 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbc63d42-9000-4c29-b2ff-2c4ef820b337" containerName="maas-api" Nov 03 22:00:08.237476 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.237290 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="884c8308-cf0b-496d-9f55-bb6563fe0238" containerName="istio-proxy" Nov 03 22:00:08.237476 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.237298 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="38287625-1740-41e8-937c-a7e55e2d8c88" containerName="istio-proxy" Nov 03 22:00:08.237476 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.237305 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ac55cb2-e271-4044-ab8e-cc5a6335af53" containerName="istio-proxy" Nov 03 22:00:08.237476 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.237311 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f2fd230-b1c2-4835-bc30-25437b149c02" containerName="istio-proxy" Nov 03 22:00:08.237476 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.237319 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f2fd230-b1c2-4835-bc30-25437b149c02" containerName="istio-proxy" Nov 03 22:00:08.237476 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.237325 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb102e63-94c5-4eed-ad81-dbb253fbc44d" containerName="payload-pre-processing" Nov 03 22:00:08.237476 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.237332 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="38287625-1740-41e8-937c-a7e55e2d8c88" containerName="istio-proxy" Nov 03 22:00:08.237476 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.237337 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ac55cb2-e271-4044-ab8e-cc5a6335af53" containerName="istio-proxy" Nov 03 22:00:08.237476 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.237343 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbc63d42-9000-4c29-b2ff-2c4ef820b337" containerName="maas-api" Nov 03 22:00:08.237476 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.237350 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b18bd0d-0a30-4471-8ced-27cce9e7b5b0" containerName="payload-processing" Nov 03 22:00:08.240760 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.240742 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" Nov 03 22:00:08.297214 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.297176 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx"] Nov 03 22:00:08.335724 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.335685 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/473b4351-c417-480c-9974-2f686f617433-tls-certs\") pod \"rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx\" (UID: \"473b4351-c417-480c-9974-2f686f617433\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" Nov 03 22:00:08.335893 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.335743 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/473b4351-c417-480c-9974-2f686f617433-home\") pod \"rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx\" (UID: \"473b4351-c417-480c-9974-2f686f617433\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" Nov 03 22:00:08.335893 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.335845 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfwmf\" (UniqueName: \"kubernetes.io/projected/473b4351-c417-480c-9974-2f686f617433-kube-api-access-jfwmf\") pod \"rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx\" (UID: \"473b4351-c417-480c-9974-2f686f617433\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" Nov 03 22:00:08.335975 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.335903 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/473b4351-c417-480c-9974-2f686f617433-tmp-dir\") pod \"rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx\" (UID: \"473b4351-c417-480c-9974-2f686f617433\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" Nov 03 22:00:08.335975 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.335967 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/473b4351-c417-480c-9974-2f686f617433-dshm\") pod \"rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx\" (UID: \"473b4351-c417-480c-9974-2f686f617433\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" Nov 03 22:00:08.336059 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.335991 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/473b4351-c417-480c-9974-2f686f617433-model-cache\") pod \"rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx\" (UID: \"473b4351-c417-480c-9974-2f686f617433\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" Nov 03 22:00:08.344309 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.344274 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-4d1799\"/\"openshift-service-ca.crt\"" Nov 03 22:00:08.344309 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.344286 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-4d1799\"/\"rate-test-model-4d1799-kserve-self-signed-certs\"" Nov 03 22:00:08.344486 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.344381 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-4d1799\"/\"default-dockercfg-f75jv\"" Nov 03 22:00:08.345893 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.345875 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-4d1799\"/\"kube-root-ca.crt\"" Nov 03 22:00:08.437068 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.437030 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfwmf\" (UniqueName: \"kubernetes.io/projected/473b4351-c417-480c-9974-2f686f617433-kube-api-access-jfwmf\") pod \"rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx\" (UID: \"473b4351-c417-480c-9974-2f686f617433\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" Nov 03 22:00:08.437279 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.437107 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/473b4351-c417-480c-9974-2f686f617433-tmp-dir\") pod \"rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx\" (UID: \"473b4351-c417-480c-9974-2f686f617433\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" Nov 03 22:00:08.437279 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.437170 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/473b4351-c417-480c-9974-2f686f617433-dshm\") pod \"rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx\" (UID: \"473b4351-c417-480c-9974-2f686f617433\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" Nov 03 22:00:08.437279 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.437190 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/473b4351-c417-480c-9974-2f686f617433-model-cache\") pod \"rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx\" (UID: \"473b4351-c417-480c-9974-2f686f617433\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" Nov 03 22:00:08.437423 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.437370 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/473b4351-c417-480c-9974-2f686f617433-tls-certs\") pod \"rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx\" (UID: \"473b4351-c417-480c-9974-2f686f617433\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" Nov 03 22:00:08.437482 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.437428 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/473b4351-c417-480c-9974-2f686f617433-home\") pod \"rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx\" (UID: \"473b4351-c417-480c-9974-2f686f617433\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" Nov 03 22:00:08.437693 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.437666 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/473b4351-c417-480c-9974-2f686f617433-tmp-dir\") pod \"rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx\" (UID: \"473b4351-c417-480c-9974-2f686f617433\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" Nov 03 22:00:08.437834 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.437697 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/473b4351-c417-480c-9974-2f686f617433-model-cache\") pod \"rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx\" (UID: \"473b4351-c417-480c-9974-2f686f617433\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" Nov 03 22:00:08.437834 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.437773 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/473b4351-c417-480c-9974-2f686f617433-home\") pod \"rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx\" (UID: \"473b4351-c417-480c-9974-2f686f617433\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" Nov 03 22:00:08.439575 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.439550 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/473b4351-c417-480c-9974-2f686f617433-dshm\") pod \"rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx\" (UID: \"473b4351-c417-480c-9974-2f686f617433\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" Nov 03 22:00:08.439892 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.439874 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/473b4351-c417-480c-9974-2f686f617433-tls-certs\") pod \"rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx\" (UID: \"473b4351-c417-480c-9974-2f686f617433\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" Nov 03 22:00:08.499884 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.499811 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfwmf\" (UniqueName: \"kubernetes.io/projected/473b4351-c417-480c-9974-2f686f617433-kube-api-access-jfwmf\") pod \"rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx\" (UID: \"473b4351-c417-480c-9974-2f686f617433\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" Nov 03 22:00:08.553454 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.553415 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" Nov 03 22:00:08.929109 ip-10-0-129-192 kubenswrapper[5615]: W1103 22:00:08.929044 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod473b4351_c417_480c_9974_2f686f617433.slice/crio-592f1dd53f8ab3c0c2f759823e4e8fd932d5453ca807fd26decbadbc2bb8aaa7 WatchSource:0}: Error finding container 592f1dd53f8ab3c0c2f759823e4e8fd932d5453ca807fd26decbadbc2bb8aaa7: Status 404 returned error can't find the container with id 592f1dd53f8ab3c0c2f759823e4e8fd932d5453ca807fd26decbadbc2bb8aaa7 Nov 03 22:00:08.931476 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:08.931448 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx"] Nov 03 22:00:09.782060 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:09.782011 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" event={"ID":"473b4351-c417-480c-9974-2f686f617433","Type":"ContainerStarted","Data":"9cf35f17d8a59750de34d2ed9a63a13eab662b7d4268973e7fc0188e96832c3b"} Nov 03 22:00:09.782531 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:09.782069 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" event={"ID":"473b4351-c417-480c-9974-2f686f617433","Type":"ContainerStarted","Data":"592f1dd53f8ab3c0c2f759823e4e8fd932d5453ca807fd26decbadbc2bb8aaa7"} Nov 03 22:00:10.553529 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:10.553494 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" Nov 03 22:00:10.619020 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:00:10.618982 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:00:18.554392 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:18.554349 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" Nov 03 22:00:18.567872 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:18.567843 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" Nov 03 22:00:18.687763 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:18.687709 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" podStartSLOduration=10.687694902 podStartE2EDuration="10.687694902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:09.909549035 +0000 UTC m=+4304.377327759" watchObservedRunningTime="2026-11-03 22:00:18.687694902 +0000 UTC m=+4313.155473624" Nov 03 22:00:18.849488 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:00:18.849401 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" Nov 03 22:00:40.619178 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:00:40.619095 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:01:10.703875 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:01:10.703814 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:01:40.620110 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:01:40.620040 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:01:57.869198 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:57.869162 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx"] Nov 03 22:01:57.869762 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:57.869748 5615 cpu_manager.go:414] "containerMap: removing container" podUID="884c8308-cf0b-496d-9f55-bb6563fe0238" containerName="istio-proxy" Nov 03 22:01:57.869810 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:57.869765 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884c8308-cf0b-496d-9f55-bb6563fe0238" containerName="istio-proxy" Nov 03 22:01:57.869810 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:57.869777 5615 cpu_manager.go:414] "containerMap: removing container" podUID="38287625-1740-41e8-937c-a7e55e2d8c88" containerName="istio-proxy" Nov 03 22:01:57.869810 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:57.869783 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38287625-1740-41e8-937c-a7e55e2d8c88" containerName="istio-proxy" Nov 03 22:01:57.869921 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:57.869911 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="884c8308-cf0b-496d-9f55-bb6563fe0238" containerName="istio-proxy" Nov 03 22:01:57.873568 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:57.873541 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" Nov 03 22:01:57.929759 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:57.929720 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx"] Nov 03 22:01:57.976687 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:57.976643 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-a6219b\"/\"kube-root-ca.crt\"" Nov 03 22:01:57.976687 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:57.976663 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-a6219b\"/\"default-dockercfg-j4qpm\"" Nov 03 22:01:57.976902 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:57.976717 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-a6219b\"/\"openshift-service-ca.crt\"" Nov 03 22:01:57.978650 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:57.978628 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-a6219b\"/\"rate-test-model-a6219b-kserve-self-signed-certs\"" Nov 03 22:01:57.988411 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:57.988376 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g2kt\" (UniqueName: \"kubernetes.io/projected/fd34f512-e1e0-433b-997f-b124daa6ded0-kube-api-access-8g2kt\") pod \"rate-test-model-a6219b-kserve-5678ccf457-xp2xx\" (UID: \"fd34f512-e1e0-433b-997f-b124daa6ded0\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" Nov 03 22:01:57.988548 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:57.988443 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd34f512-e1e0-433b-997f-b124daa6ded0-dshm\") pod \"rate-test-model-a6219b-kserve-5678ccf457-xp2xx\" (UID: \"fd34f512-e1e0-433b-997f-b124daa6ded0\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" Nov 03 22:01:57.988592 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:57.988544 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd34f512-e1e0-433b-997f-b124daa6ded0-model-cache\") pod \"rate-test-model-a6219b-kserve-5678ccf457-xp2xx\" (UID: \"fd34f512-e1e0-433b-997f-b124daa6ded0\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" Nov 03 22:01:57.988592 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:57.988577 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd34f512-e1e0-433b-997f-b124daa6ded0-tmp-dir\") pod \"rate-test-model-a6219b-kserve-5678ccf457-xp2xx\" (UID: \"fd34f512-e1e0-433b-997f-b124daa6ded0\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" Nov 03 22:01:57.988721 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:57.988704 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd34f512-e1e0-433b-997f-b124daa6ded0-home\") pod \"rate-test-model-a6219b-kserve-5678ccf457-xp2xx\" (UID: \"fd34f512-e1e0-433b-997f-b124daa6ded0\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" Nov 03 22:01:57.988781 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:57.988768 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd34f512-e1e0-433b-997f-b124daa6ded0-tls-certs\") pod \"rate-test-model-a6219b-kserve-5678ccf457-xp2xx\" (UID: \"fd34f512-e1e0-433b-997f-b124daa6ded0\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" Nov 03 22:01:58.089809 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:58.089765 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd34f512-e1e0-433b-997f-b124daa6ded0-home\") pod \"rate-test-model-a6219b-kserve-5678ccf457-xp2xx\" (UID: \"fd34f512-e1e0-433b-997f-b124daa6ded0\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" Nov 03 22:01:58.090011 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:58.089829 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd34f512-e1e0-433b-997f-b124daa6ded0-tls-certs\") pod \"rate-test-model-a6219b-kserve-5678ccf457-xp2xx\" (UID: \"fd34f512-e1e0-433b-997f-b124daa6ded0\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" Nov 03 22:01:58.090011 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:58.089936 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g2kt\" (UniqueName: \"kubernetes.io/projected/fd34f512-e1e0-433b-997f-b124daa6ded0-kube-api-access-8g2kt\") pod \"rate-test-model-a6219b-kserve-5678ccf457-xp2xx\" (UID: \"fd34f512-e1e0-433b-997f-b124daa6ded0\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" Nov 03 22:01:58.090011 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:58.089972 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd34f512-e1e0-433b-997f-b124daa6ded0-dshm\") pod \"rate-test-model-a6219b-kserve-5678ccf457-xp2xx\" (UID: \"fd34f512-e1e0-433b-997f-b124daa6ded0\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" Nov 03 22:01:58.090207 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:58.090013 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd34f512-e1e0-433b-997f-b124daa6ded0-model-cache\") pod \"rate-test-model-a6219b-kserve-5678ccf457-xp2xx\" (UID: \"fd34f512-e1e0-433b-997f-b124daa6ded0\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" Nov 03 22:01:58.090207 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:58.090031 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd34f512-e1e0-433b-997f-b124daa6ded0-tmp-dir\") pod \"rate-test-model-a6219b-kserve-5678ccf457-xp2xx\" (UID: \"fd34f512-e1e0-433b-997f-b124daa6ded0\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" Nov 03 22:01:58.090434 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:58.090416 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd34f512-e1e0-433b-997f-b124daa6ded0-tmp-dir\") pod \"rate-test-model-a6219b-kserve-5678ccf457-xp2xx\" (UID: \"fd34f512-e1e0-433b-997f-b124daa6ded0\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" Nov 03 22:01:58.090607 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:58.090591 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd34f512-e1e0-433b-997f-b124daa6ded0-home\") pod \"rate-test-model-a6219b-kserve-5678ccf457-xp2xx\" (UID: \"fd34f512-e1e0-433b-997f-b124daa6ded0\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" Nov 03 22:01:58.091195 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:58.091165 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd34f512-e1e0-433b-997f-b124daa6ded0-model-cache\") pod \"rate-test-model-a6219b-kserve-5678ccf457-xp2xx\" (UID: \"fd34f512-e1e0-433b-997f-b124daa6ded0\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" Nov 03 22:01:58.093351 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:58.093322 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd34f512-e1e0-433b-997f-b124daa6ded0-tls-certs\") pod \"rate-test-model-a6219b-kserve-5678ccf457-xp2xx\" (UID: \"fd34f512-e1e0-433b-997f-b124daa6ded0\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" Nov 03 22:01:58.093718 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:58.093696 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd34f512-e1e0-433b-997f-b124daa6ded0-dshm\") pod \"rate-test-model-a6219b-kserve-5678ccf457-xp2xx\" (UID: \"fd34f512-e1e0-433b-997f-b124daa6ded0\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" Nov 03 22:01:58.153504 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:58.153424 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g2kt\" (UniqueName: \"kubernetes.io/projected/fd34f512-e1e0-433b-997f-b124daa6ded0-kube-api-access-8g2kt\") pod \"rate-test-model-a6219b-kserve-5678ccf457-xp2xx\" (UID: \"fd34f512-e1e0-433b-997f-b124daa6ded0\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" Nov 03 22:01:58.187130 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:58.187062 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" Nov 03 22:01:58.566468 ip-10-0-129-192 kubenswrapper[5615]: W1103 22:01:58.566435 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd34f512_e1e0_433b_997f_b124daa6ded0.slice/crio-46a4aa88f79043a8ae9b980aadd58d726f8030271d2eb2e51e7a41e79b3439e5 WatchSource:0}: Error finding container 46a4aa88f79043a8ae9b980aadd58d726f8030271d2eb2e51e7a41e79b3439e5: Status 404 returned error can't find the container with id 46a4aa88f79043a8ae9b980aadd58d726f8030271d2eb2e51e7a41e79b3439e5 Nov 03 22:01:58.568895 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:58.568867 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx"] Nov 03 22:01:58.590880 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:58.590837 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" event={"ID":"fd34f512-e1e0-433b-997f-b124daa6ded0","Type":"ContainerStarted","Data":"46a4aa88f79043a8ae9b980aadd58d726f8030271d2eb2e51e7a41e79b3439e5"} Nov 03 22:01:59.599088 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:59.599045 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" event={"ID":"fd34f512-e1e0-433b-997f-b124daa6ded0","Type":"ContainerStarted","Data":"e8372699dcdf0287dbaaf6f2451adbba07ee24e494812c888d1d6cc77db9ffa8"} Nov 03 22:01:59.721812 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:01:59.721754 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" podStartSLOduration=2.721739247 podStartE2EDuration="2.721739247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:01:59.719767362 +0000 UTC m=+4414.187546084" watchObservedRunningTime="2026-11-03 22:01:59.721739247 +0000 UTC m=+4414.189517969" Nov 03 22:02:00.187728 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:02:00.187616 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" Nov 03 22:02:08.187515 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:02:08.187420 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" Nov 03 22:02:08.200437 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:02:08.200404 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" Nov 03 22:02:08.665203 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:02:08.665174 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" Nov 03 22:02:10.619758 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:02:10.619723 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:02:40.619681 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:02:40.619641 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:03:10.619873 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:03:10.619829 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:03:26.750625 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:26.750588 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-c11538-d9df47586-gj7tm"] Nov 03 22:03:31.751210 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:31.751140 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c11538-d9df47586-gj7tm" podUID="5c90b432-31d5-49c0-8f4e-11397476f868" containerName="payload-processing" containerID="cri-o://f703584d4f1b329553ae2e4e438c1ca884ed72e9c5e571f1c8e21c7414a50947" gracePeriod=25 Nov 03 22:03:32.038343 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:32.038313 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c11538-d9df47586-gj7tm" Nov 03 22:03:32.211453 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:32.211420 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c90b432-31d5-49c0-8f4e-11397476f868-kube-api-access-kdfkf\" (UniqueName: \"kubernetes.io/projected/5c90b432-31d5-49c0-8f4e-11397476f868-kube-api-access-kdfkf\") pod \"5c90b432-31d5-49c0-8f4e-11397476f868\" (UID: \"5c90b432-31d5-49c0-8f4e-11397476f868\") " Nov 03 22:03:32.211636 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:32.211549 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c90b432-31d5-49c0-8f4e-11397476f868-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c90b432-31d5-49c0-8f4e-11397476f868-plugins-config-volume\") pod \"5c90b432-31d5-49c0-8f4e-11397476f868\" (UID: \"5c90b432-31d5-49c0-8f4e-11397476f868\") " Nov 03 22:03:32.211918 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:32.211885 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c90b432-31d5-49c0-8f4e-11397476f868-plugins-config-volume" pod "5c90b432-31d5-49c0-8f4e-11397476f868" (UID: "5c90b432-31d5-49c0-8f4e-11397476f868"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:03:32.212196 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:32.212177 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c90b432-31d5-49c0-8f4e-11397476f868-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:32.213563 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:32.213538 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c90b432-31d5-49c0-8f4e-11397476f868-kube-api-access-kdfkf" pod "5c90b432-31d5-49c0-8f4e-11397476f868" (UID: "5c90b432-31d5-49c0-8f4e-11397476f868"). InnerVolumeSpecName "kube-api-access-kdfkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:03:32.285983 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:32.285898 5615 generic.go:356] "Generic (PLEG): container finished" podID="5c90b432-31d5-49c0-8f4e-11397476f868" containerID="f703584d4f1b329553ae2e4e438c1ca884ed72e9c5e571f1c8e21c7414a50947" exitCode=0 Nov 03 22:03:32.285983 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:32.285943 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c11538-d9df47586-gj7tm" Nov 03 22:03:32.286263 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:32.285977 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c11538-d9df47586-gj7tm" event={"ID":"5c90b432-31d5-49c0-8f4e-11397476f868","Type":"ContainerDied","Data":"f703584d4f1b329553ae2e4e438c1ca884ed72e9c5e571f1c8e21c7414a50947"} Nov 03 22:03:32.286263 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:32.286032 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c11538-d9df47586-gj7tm" event={"ID":"5c90b432-31d5-49c0-8f4e-11397476f868","Type":"ContainerDied","Data":"ae912b5668d7966c10084c7d211d3761b85fe8af3127e276ead5057c73282ce1"} Nov 03 22:03:32.286263 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:32.286056 5615 scope.go:122] "RemoveContainer" containerID="f703584d4f1b329553ae2e4e438c1ca884ed72e9c5e571f1c8e21c7414a50947" Nov 03 22:03:32.296612 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:32.296587 5615 scope.go:122] "RemoveContainer" containerID="f703584d4f1b329553ae2e4e438c1ca884ed72e9c5e571f1c8e21c7414a50947" Nov 03 22:03:32.296889 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:03:32.296864 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f703584d4f1b329553ae2e4e438c1ca884ed72e9c5e571f1c8e21c7414a50947\": container with ID starting with f703584d4f1b329553ae2e4e438c1ca884ed72e9c5e571f1c8e21c7414a50947 not found: ID does not exist" containerID="f703584d4f1b329553ae2e4e438c1ca884ed72e9c5e571f1c8e21c7414a50947" Nov 03 22:03:32.296977 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:32.296897 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f703584d4f1b329553ae2e4e438c1ca884ed72e9c5e571f1c8e21c7414a50947"} err="failed to get container status \"f703584d4f1b329553ae2e4e438c1ca884ed72e9c5e571f1c8e21c7414a50947\": rpc error: code = NotFound desc = could not find container \"f703584d4f1b329553ae2e4e438c1ca884ed72e9c5e571f1c8e21c7414a50947\": container with ID starting with f703584d4f1b329553ae2e4e438c1ca884ed72e9c5e571f1c8e21c7414a50947 not found: ID does not exist" Nov 03 22:03:32.313517 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:32.313488 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdfkf\" (UniqueName: \"kubernetes.io/projected/5c90b432-31d5-49c0-8f4e-11397476f868-kube-api-access-kdfkf\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:32.454247 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:32.454207 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-c11538-d9df47586-gj7tm"] Nov 03 22:03:32.458214 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:32.458183 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-c11538-d9df47586-gj7tm"] Nov 03 22:03:33.653325 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:33.653293 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c90b432-31d5-49c0-8f4e-11397476f868" path="/var/lib/kubelet/pods/5c90b432-31d5-49c0-8f4e-11397476f868/volumes" Nov 03 22:03:39.245658 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:39.245579 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt"] Nov 03 22:03:39.246086 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:39.245798 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" podUID="625d3189-ee5d-4264-b3a2-469ddecb4427" containerName="istio-proxy" containerID="cri-o://5f359253c9aa77816a535686cd2bbbfd19cff1a7ddc321b65652ed2fd2d4426a" gracePeriod=30 Nov 03 22:03:40.619131 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:03:40.619062 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:03:44.368806 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:44.368775 5615 generic.go:356] "Generic (PLEG): container finished" podID="625d3189-ee5d-4264-b3a2-469ddecb4427" containerID="5f359253c9aa77816a535686cd2bbbfd19cff1a7ddc321b65652ed2fd2d4426a" exitCode=0 Nov 03 22:03:44.369144 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:44.368846 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" event={"ID":"625d3189-ee5d-4264-b3a2-469ddecb4427","Type":"ContainerDied","Data":"5f359253c9aa77816a535686cd2bbbfd19cff1a7ddc321b65652ed2fd2d4426a"} Nov 03 22:03:44.518666 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:44.518607 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 22:03:44.574164 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:44.574113 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/625d3189-ee5d-4264-b3a2-469ddecb4427-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625d3189-ee5d-4264-b3a2-469ddecb4427-istio-podinfo\") pod \"625d3189-ee5d-4264-b3a2-469ddecb4427\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " Nov 03 22:03:44.574350 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:44.574215 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-workload-certs\") pod \"625d3189-ee5d-4264-b3a2-469ddecb4427\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " Nov 03 22:03:44.574350 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:44.574290 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-credential-socket\") pod \"625d3189-ee5d-4264-b3a2-469ddecb4427\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " Nov 03 22:03:44.574438 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:44.574372 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/625d3189-ee5d-4264-b3a2-469ddecb4427-istio-token\" (UniqueName: \"kubernetes.io/projected/625d3189-ee5d-4264-b3a2-469ddecb4427-istio-token\") pod \"625d3189-ee5d-4264-b3a2-469ddecb4427\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " Nov 03 22:03:44.574438 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:44.574405 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/625d3189-ee5d-4264-b3a2-469ddecb4427-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625d3189-ee5d-4264-b3a2-469ddecb4427-istiod-ca-cert\") pod \"625d3189-ee5d-4264-b3a2-469ddecb4427\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " Nov 03 22:03:44.574549 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:44.574433 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-istio-envoy\") pod \"625d3189-ee5d-4264-b3a2-469ddecb4427\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " Nov 03 22:03:44.574549 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:44.574516 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-istio-data\") pod \"625d3189-ee5d-4264-b3a2-469ddecb4427\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " Nov 03 22:03:44.574713 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:44.574539 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-workload-certs" pod "625d3189-ee5d-4264-b3a2-469ddecb4427" (UID: "625d3189-ee5d-4264-b3a2-469ddecb4427"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:03:44.574713 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:44.574566 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-workload-socket\") pod \"625d3189-ee5d-4264-b3a2-469ddecb4427\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " Nov 03 22:03:44.574713 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:44.574594 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-credential-socket" pod "625d3189-ee5d-4264-b3a2-469ddecb4427" (UID: "625d3189-ee5d-4264-b3a2-469ddecb4427"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:03:44.574713 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:44.574640 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/625d3189-ee5d-4264-b3a2-469ddecb4427-kube-api-access-rc5th\" (UniqueName: \"kubernetes.io/projected/625d3189-ee5d-4264-b3a2-469ddecb4427-kube-api-access-rc5th\") pod \"625d3189-ee5d-4264-b3a2-469ddecb4427\" (UID: \"625d3189-ee5d-4264-b3a2-469ddecb4427\") " Nov 03 22:03:44.574940 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:44.574888 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-istio-data" pod "625d3189-ee5d-4264-b3a2-469ddecb4427" (UID: "625d3189-ee5d-4264-b3a2-469ddecb4427"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:03:44.574940 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:44.574916 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/625d3189-ee5d-4264-b3a2-469ddecb4427-istiod-ca-cert" pod "625d3189-ee5d-4264-b3a2-469ddecb4427" (UID: "625d3189-ee5d-4264-b3a2-469ddecb4427"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:03:44.575194 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:44.575168 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-workload-socket" pod "625d3189-ee5d-4264-b3a2-469ddecb4427" (UID: "625d3189-ee5d-4264-b3a2-469ddecb4427"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:03:44.575388 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:44.575349 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:44.575388 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:44.575372 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625d3189-ee5d-4264-b3a2-469ddecb4427-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:44.575556 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:44.575387 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:44.575556 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:44.575400 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:44.575556 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:44.575408 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:44.576719 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:44.576692 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-istio-envoy" pod "625d3189-ee5d-4264-b3a2-469ddecb4427" (UID: "625d3189-ee5d-4264-b3a2-469ddecb4427"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:03:44.576719 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:44.576700 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625d3189-ee5d-4264-b3a2-469ddecb4427-istio-token" pod "625d3189-ee5d-4264-b3a2-469ddecb4427" (UID: "625d3189-ee5d-4264-b3a2-469ddecb4427"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:03:44.576884 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:44.576865 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/625d3189-ee5d-4264-b3a2-469ddecb4427-istio-podinfo" pod "625d3189-ee5d-4264-b3a2-469ddecb4427" (UID: "625d3189-ee5d-4264-b3a2-469ddecb4427"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:03:44.577108 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:44.577069 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625d3189-ee5d-4264-b3a2-469ddecb4427-kube-api-access-rc5th" pod "625d3189-ee5d-4264-b3a2-469ddecb4427" (UID: "625d3189-ee5d-4264-b3a2-469ddecb4427"). InnerVolumeSpecName "kube-api-access-rc5th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:03:44.676658 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:44.676623 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625d3189-ee5d-4264-b3a2-469ddecb4427-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:44.676658 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:44.676655 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625d3189-ee5d-4264-b3a2-469ddecb4427-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:44.676658 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:44.676665 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc5th\" (UniqueName: \"kubernetes.io/projected/625d3189-ee5d-4264-b3a2-469ddecb4427-kube-api-access-rc5th\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:44.676900 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:44.676675 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625d3189-ee5d-4264-b3a2-469ddecb4427-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:45.377297 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:45.377262 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" Nov 03 22:03:45.377791 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:45.377292 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt" event={"ID":"625d3189-ee5d-4264-b3a2-469ddecb4427","Type":"ContainerDied","Data":"2aa4131b174cf6f3f0d15177deb1762b8490199bf847e4c2ed7b760107b39d10"} Nov 03 22:03:45.377791 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:45.377356 5615 scope.go:122] "RemoveContainer" containerID="5f359253c9aa77816a535686cd2bbbfd19cff1a7ddc321b65652ed2fd2d4426a" Nov 03 22:03:45.596655 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:45.596617 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt"] Nov 03 22:03:45.600761 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:45.600732 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mncxt"] Nov 03 22:03:45.651634 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:45.651556 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="625d3189-ee5d-4264-b3a2-469ddecb4427" path="/var/lib/kubelet/pods/625d3189-ee5d-4264-b3a2-469ddecb4427/volumes" Nov 03 22:03:55.724463 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:55.724426 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8"] Nov 03 22:03:55.724971 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:55.724695 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8" podUID="98dbffe1-fe8a-43fa-821a-cfe49074a6a9" containerName="maas-api" containerID="cri-o://b3b750da0a6dd7410cf5146d9ceb9b4c4ca2b9b3b352cb94db78e728cc203930" gracePeriod=30 Nov 03 22:03:56.038729 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:56.038662 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8" Nov 03 22:03:56.146570 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:56.146520 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-openshift-service-ca\") pod \"98dbffe1-fe8a-43fa-821a-cfe49074a6a9\" (UID: \"98dbffe1-fe8a-43fa-821a-cfe49074a6a9\") " Nov 03 22:03:56.150590 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:56.150554 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-metrics-certs\") pod \"98dbffe1-fe8a-43fa-821a-cfe49074a6a9\" (UID: \"98dbffe1-fe8a-43fa-821a-cfe49074a6a9\") " Nov 03 22:03:56.154911 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:56.150328 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-openshift-service-ca" pod "98dbffe1-fe8a-43fa-821a-cfe49074a6a9" (UID: "98dbffe1-fe8a-43fa-821a-cfe49074a6a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:03:56.156491 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:56.155181 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-metrics-certs" pod "98dbffe1-fe8a-43fa-821a-cfe49074a6a9" (UID: "98dbffe1-fe8a-43fa-821a-cfe49074a6a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:03:56.156491 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:56.155356 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-kube-api-access-hlbc7\" (UniqueName: \"kubernetes.io/projected/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-kube-api-access-hlbc7\") pod \"98dbffe1-fe8a-43fa-821a-cfe49074a6a9\" (UID: \"98dbffe1-fe8a-43fa-821a-cfe49074a6a9\") " Nov 03 22:03:56.156491 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:56.155411 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-maas-api-tls\") pod \"98dbffe1-fe8a-43fa-821a-cfe49074a6a9\" (UID: \"98dbffe1-fe8a-43fa-821a-cfe49074a6a9\") " Nov 03 22:03:56.156491 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:56.156061 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:56.156491 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:56.156100 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:56.195040 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:56.194567 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-kube-api-access-hlbc7" pod "98dbffe1-fe8a-43fa-821a-cfe49074a6a9" (UID: "98dbffe1-fe8a-43fa-821a-cfe49074a6a9"). InnerVolumeSpecName "kube-api-access-hlbc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:03:56.199579 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:56.199503 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-maas-api-tls" pod "98dbffe1-fe8a-43fa-821a-cfe49074a6a9" (UID: "98dbffe1-fe8a-43fa-821a-cfe49074a6a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:03:56.257404 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:56.257321 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlbc7\" (UniqueName: \"kubernetes.io/projected/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-kube-api-access-hlbc7\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:56.257404 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:56.257351 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98dbffe1-fe8a-43fa-821a-cfe49074a6a9-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:56.484045 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:56.484012 5615 generic.go:356] "Generic (PLEG): container finished" podID="98dbffe1-fe8a-43fa-821a-cfe49074a6a9" containerID="b3b750da0a6dd7410cf5146d9ceb9b4c4ca2b9b3b352cb94db78e728cc203930" exitCode=0 Nov 03 22:03:56.484258 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:56.484057 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8" Nov 03 22:03:56.484258 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:56.484100 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8" event={"ID":"98dbffe1-fe8a-43fa-821a-cfe49074a6a9","Type":"ContainerDied","Data":"b3b750da0a6dd7410cf5146d9ceb9b4c4ca2b9b3b352cb94db78e728cc203930"} Nov 03 22:03:56.484258 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:56.484156 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8" event={"ID":"98dbffe1-fe8a-43fa-821a-cfe49074a6a9","Type":"ContainerDied","Data":"93bfd54cb7183d9b957570e35b1a8fff28131d80e643cf3f1c699de130affffe"} Nov 03 22:03:56.484258 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:56.484177 5615 scope.go:122] "RemoveContainer" containerID="b3b750da0a6dd7410cf5146d9ceb9b4c4ca2b9b3b352cb94db78e728cc203930" Nov 03 22:03:56.495456 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:56.495430 5615 scope.go:122] "RemoveContainer" containerID="b3b750da0a6dd7410cf5146d9ceb9b4c4ca2b9b3b352cb94db78e728cc203930" Nov 03 22:03:56.495737 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:03:56.495714 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3b750da0a6dd7410cf5146d9ceb9b4c4ca2b9b3b352cb94db78e728cc203930\": container with ID starting with b3b750da0a6dd7410cf5146d9ceb9b4c4ca2b9b3b352cb94db78e728cc203930 not found: ID does not exist" containerID="b3b750da0a6dd7410cf5146d9ceb9b4c4ca2b9b3b352cb94db78e728cc203930" Nov 03 22:03:56.495846 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:56.495746 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3b750da0a6dd7410cf5146d9ceb9b4c4ca2b9b3b352cb94db78e728cc203930"} err="failed to get container status \"b3b750da0a6dd7410cf5146d9ceb9b4c4ca2b9b3b352cb94db78e728cc203930\": rpc error: code = NotFound desc = could not find container \"b3b750da0a6dd7410cf5146d9ceb9b4c4ca2b9b3b352cb94db78e728cc203930\": container with ID starting with b3b750da0a6dd7410cf5146d9ceb9b4c4ca2b9b3b352cb94db78e728cc203930 not found: ID does not exist" Nov 03 22:03:56.652468 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:56.652388 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8"] Nov 03 22:03:56.731324 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:56.731269 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c11538-6db9cc899f-wt2l8"] Nov 03 22:03:57.656653 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:03:57.656614 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98dbffe1-fe8a-43fa-821a-cfe49074a6a9" path="/var/lib/kubelet/pods/98dbffe1-fe8a-43fa-821a-cfe49074a6a9/volumes" Nov 03 22:04:00.337597 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:00.337560 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k"] Nov 03 22:04:00.338235 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:00.337827 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" podUID="77818da7-0c48-434f-b2cb-68d47b3c2a33" containerName="istio-proxy" containerID="cri-o://f1f32e6c781ccbef7c25972505e586d06624ce0830692fff71321ec8006e0d00" gracePeriod=30 Nov 03 22:04:03.067736 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:03.067693 5615 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:04:03.068183 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:03.067772 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" podUID="77818da7-0c48-434f-b2cb-68d47b3c2a33" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:04:05.561964 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.561924 5615 generic.go:356] "Generic (PLEG): container finished" podID="77818da7-0c48-434f-b2cb-68d47b3c2a33" containerID="f1f32e6c781ccbef7c25972505e586d06624ce0830692fff71321ec8006e0d00" exitCode=0 Nov 03 22:04:05.562463 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.561996 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" event={"ID":"77818da7-0c48-434f-b2cb-68d47b3c2a33","Type":"ContainerDied","Data":"f1f32e6c781ccbef7c25972505e586d06624ce0830692fff71321ec8006e0d00"} Nov 03 22:04:05.562463 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.562088 5615 scope.go:122] "RemoveContainer" containerID="be7edeb286d1dda785653b4dcfada1800ead4f1fc7961349c0f7246bd05b8f58" Nov 03 22:04:05.617889 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.617828 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 22:04:05.786880 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.786849 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-istio-data\") pod \"77818da7-0c48-434f-b2cb-68d47b3c2a33\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " Nov 03 22:04:05.787133 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.786907 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77818da7-0c48-434f-b2cb-68d47b3c2a33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77818da7-0c48-434f-b2cb-68d47b3c2a33-istio-podinfo\") pod \"77818da7-0c48-434f-b2cb-68d47b3c2a33\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " Nov 03 22:04:05.787133 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.786928 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-workload-certs\") pod \"77818da7-0c48-434f-b2cb-68d47b3c2a33\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " Nov 03 22:04:05.787133 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.786974 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77818da7-0c48-434f-b2cb-68d47b3c2a33-kube-api-access-nv29b\" (UniqueName: \"kubernetes.io/projected/77818da7-0c48-434f-b2cb-68d47b3c2a33-kube-api-access-nv29b\") pod \"77818da7-0c48-434f-b2cb-68d47b3c2a33\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " Nov 03 22:04:05.787133 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.787052 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77818da7-0c48-434f-b2cb-68d47b3c2a33-istio-token\" (UniqueName: \"kubernetes.io/projected/77818da7-0c48-434f-b2cb-68d47b3c2a33-istio-token\") pod \"77818da7-0c48-434f-b2cb-68d47b3c2a33\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " Nov 03 22:04:05.787133 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.787096 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-istio-data" pod "77818da7-0c48-434f-b2cb-68d47b3c2a33" (UID: "77818da7-0c48-434f-b2cb-68d47b3c2a33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:04:05.787133 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.787105 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-istio-envoy\") pod \"77818da7-0c48-434f-b2cb-68d47b3c2a33\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " Nov 03 22:04:05.787485 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.787189 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-workload-socket\") pod \"77818da7-0c48-434f-b2cb-68d47b3c2a33\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " Nov 03 22:04:05.787485 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.787280 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-workload-certs" pod "77818da7-0c48-434f-b2cb-68d47b3c2a33" (UID: "77818da7-0c48-434f-b2cb-68d47b3c2a33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:04:05.787485 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.787291 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77818da7-0c48-434f-b2cb-68d47b3c2a33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77818da7-0c48-434f-b2cb-68d47b3c2a33-istiod-ca-cert\") pod \"77818da7-0c48-434f-b2cb-68d47b3c2a33\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " Nov 03 22:04:05.787485 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.787386 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-credential-socket\") pod \"77818da7-0c48-434f-b2cb-68d47b3c2a33\" (UID: \"77818da7-0c48-434f-b2cb-68d47b3c2a33\") " Nov 03 22:04:05.787744 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.787621 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-workload-socket" pod "77818da7-0c48-434f-b2cb-68d47b3c2a33" (UID: "77818da7-0c48-434f-b2cb-68d47b3c2a33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:04:05.787814 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.787776 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-credential-socket" pod "77818da7-0c48-434f-b2cb-68d47b3c2a33" (UID: "77818da7-0c48-434f-b2cb-68d47b3c2a33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:04:05.789029 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.788250 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77818da7-0c48-434f-b2cb-68d47b3c2a33-istiod-ca-cert" pod "77818da7-0c48-434f-b2cb-68d47b3c2a33" (UID: "77818da7-0c48-434f-b2cb-68d47b3c2a33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:04:05.789029 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.788360 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:05.789029 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.788395 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:05.789029 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.788411 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:05.789029 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.788426 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:05.789932 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.789903 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77818da7-0c48-434f-b2cb-68d47b3c2a33-kube-api-access-nv29b" pod "77818da7-0c48-434f-b2cb-68d47b3c2a33" (UID: "77818da7-0c48-434f-b2cb-68d47b3c2a33"). InnerVolumeSpecName "kube-api-access-nv29b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:04:05.790621 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.790513 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77818da7-0c48-434f-b2cb-68d47b3c2a33-istio-podinfo" pod "77818da7-0c48-434f-b2cb-68d47b3c2a33" (UID: "77818da7-0c48-434f-b2cb-68d47b3c2a33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:04:05.790621 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.790597 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77818da7-0c48-434f-b2cb-68d47b3c2a33-istio-token" pod "77818da7-0c48-434f-b2cb-68d47b3c2a33" (UID: "77818da7-0c48-434f-b2cb-68d47b3c2a33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:04:05.790781 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.790598 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-istio-envoy" pod "77818da7-0c48-434f-b2cb-68d47b3c2a33" (UID: "77818da7-0c48-434f-b2cb-68d47b3c2a33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:04:05.889629 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.889541 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77818da7-0c48-434f-b2cb-68d47b3c2a33-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:05.889629 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.889574 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77818da7-0c48-434f-b2cb-68d47b3c2a33-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:05.889629 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.889584 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77818da7-0c48-434f-b2cb-68d47b3c2a33-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:05.889629 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.889594 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77818da7-0c48-434f-b2cb-68d47b3c2a33-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:05.889629 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:05.889604 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv29b\" (UniqueName: \"kubernetes.io/projected/77818da7-0c48-434f-b2cb-68d47b3c2a33-kube-api-access-nv29b\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:06.569765 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:06.569736 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" Nov 03 22:04:06.570223 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:06.569777 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k" event={"ID":"77818da7-0c48-434f-b2cb-68d47b3c2a33","Type":"ContainerDied","Data":"c484cf5692862b0208164eec64362a8eee9d82fd3463780b9fc31a17f886f93f"} Nov 03 22:04:06.570223 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:06.569839 5615 scope.go:122] "RemoveContainer" containerID="f1f32e6c781ccbef7c25972505e586d06624ce0830692fff71321ec8006e0d00" Nov 03 22:04:06.739065 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:06.739027 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k"] Nov 03 22:04:06.790339 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:06.790300 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-4744k"] Nov 03 22:04:07.651546 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:07.651508 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77818da7-0c48-434f-b2cb-68d47b3c2a33" path="/var/lib/kubelet/pods/77818da7-0c48-434f-b2cb-68d47b3c2a33/volumes" Nov 03 22:04:10.619759 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:04:10.619723 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:04:20.103547 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:20.103509 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-320f60-57b9576c9-7frrd"] Nov 03 22:04:25.103987 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:25.103946 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-320f60-57b9576c9-7frrd" podUID="f3f0e38e-e1e7-451d-acc2-26df86d3c8ac" containerName="payload-pre-processing" containerID="cri-o://7196992cdbae6edaa1b0a98b4cfe85686852175aa616f7d44449e2a7e9a35445" gracePeriod=25 Nov 03 22:04:25.359392 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:25.359323 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-320f60-57b9576c9-7frrd" Nov 03 22:04:25.441539 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:25.441494 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f0e38e-e1e7-451d-acc2-26df86d3c8ac-kube-api-access-8hchm\" (UniqueName: \"kubernetes.io/projected/f3f0e38e-e1e7-451d-acc2-26df86d3c8ac-kube-api-access-8hchm\") pod \"f3f0e38e-e1e7-451d-acc2-26df86d3c8ac\" (UID: \"f3f0e38e-e1e7-451d-acc2-26df86d3c8ac\") " Nov 03 22:04:25.441734 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:25.441548 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3f0e38e-e1e7-451d-acc2-26df86d3c8ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3f0e38e-e1e7-451d-acc2-26df86d3c8ac-plugins-config-volume\") pod \"f3f0e38e-e1e7-451d-acc2-26df86d3c8ac\" (UID: \"f3f0e38e-e1e7-451d-acc2-26df86d3c8ac\") " Nov 03 22:04:25.441949 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:25.441923 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f0e38e-e1e7-451d-acc2-26df86d3c8ac-plugins-config-volume" pod "f3f0e38e-e1e7-451d-acc2-26df86d3c8ac" (UID: "f3f0e38e-e1e7-451d-acc2-26df86d3c8ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:04:25.443671 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:25.443650 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f0e38e-e1e7-451d-acc2-26df86d3c8ac-kube-api-access-8hchm" pod "f3f0e38e-e1e7-451d-acc2-26df86d3c8ac" (UID: "f3f0e38e-e1e7-451d-acc2-26df86d3c8ac"). InnerVolumeSpecName "kube-api-access-8hchm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:04:25.543014 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:25.542971 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hchm\" (UniqueName: \"kubernetes.io/projected/f3f0e38e-e1e7-451d-acc2-26df86d3c8ac-kube-api-access-8hchm\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:25.543014 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:25.543007 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3f0e38e-e1e7-451d-acc2-26df86d3c8ac-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:25.692628 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:25.692591 5615 generic.go:356] "Generic (PLEG): container finished" podID="f3f0e38e-e1e7-451d-acc2-26df86d3c8ac" containerID="7196992cdbae6edaa1b0a98b4cfe85686852175aa616f7d44449e2a7e9a35445" exitCode=0 Nov 03 22:04:25.692804 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:25.692636 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-320f60-57b9576c9-7frrd" Nov 03 22:04:25.692804 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:25.692658 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-320f60-57b9576c9-7frrd" event={"ID":"f3f0e38e-e1e7-451d-acc2-26df86d3c8ac","Type":"ContainerDied","Data":"7196992cdbae6edaa1b0a98b4cfe85686852175aa616f7d44449e2a7e9a35445"} Nov 03 22:04:25.692804 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:25.692698 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-320f60-57b9576c9-7frrd" event={"ID":"f3f0e38e-e1e7-451d-acc2-26df86d3c8ac","Type":"ContainerDied","Data":"b6ce032235a2c3ba2c3b586f8940a5b45386cb4b231f50ca067d930c5539d68d"} Nov 03 22:04:25.692804 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:25.692718 5615 scope.go:122] "RemoveContainer" containerID="7196992cdbae6edaa1b0a98b4cfe85686852175aa616f7d44449e2a7e9a35445" Nov 03 22:04:25.702315 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:25.702293 5615 scope.go:122] "RemoveContainer" containerID="7196992cdbae6edaa1b0a98b4cfe85686852175aa616f7d44449e2a7e9a35445" Nov 03 22:04:25.702592 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:04:25.702568 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7196992cdbae6edaa1b0a98b4cfe85686852175aa616f7d44449e2a7e9a35445\": container with ID starting with 7196992cdbae6edaa1b0a98b4cfe85686852175aa616f7d44449e2a7e9a35445 not found: ID does not exist" containerID="7196992cdbae6edaa1b0a98b4cfe85686852175aa616f7d44449e2a7e9a35445" Nov 03 22:04:25.702659 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:25.702605 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7196992cdbae6edaa1b0a98b4cfe85686852175aa616f7d44449e2a7e9a35445"} err="failed to get container status \"7196992cdbae6edaa1b0a98b4cfe85686852175aa616f7d44449e2a7e9a35445\": rpc error: code = NotFound desc = could not find container \"7196992cdbae6edaa1b0a98b4cfe85686852175aa616f7d44449e2a7e9a35445\": container with ID starting with 7196992cdbae6edaa1b0a98b4cfe85686852175aa616f7d44449e2a7e9a35445 not found: ID does not exist" Nov 03 22:04:25.867117 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:25.867046 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-320f60-57b9576c9-7frrd"] Nov 03 22:04:25.871998 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:25.871967 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-320f60-57b9576c9-7frrd"] Nov 03 22:04:27.651199 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:27.651158 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3f0e38e-e1e7-451d-acc2-26df86d3c8ac" path="/var/lib/kubelet/pods/f3f0e38e-e1e7-451d-acc2-26df86d3c8ac/volumes" Nov 03 22:04:39.138657 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:39.138617 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2"] Nov 03 22:04:39.139569 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:39.138833 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2" podUID="933ca863-b42a-4f84-9d67-b36634e0d9b4" containerName="maas-api" containerID="cri-o://dff160c8f699bd8be9082207033d5790be32035bd1d808d3d698cbd0ac178842" gracePeriod=30 Nov 03 22:04:39.428751 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:39.428717 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2" Nov 03 22:04:39.538421 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:39.535972 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/933ca863-b42a-4f84-9d67-b36634e0d9b4-kube-api-access-j4dvt\" (UniqueName: \"kubernetes.io/projected/933ca863-b42a-4f84-9d67-b36634e0d9b4-kube-api-access-j4dvt\") pod \"933ca863-b42a-4f84-9d67-b36634e0d9b4\" (UID: \"933ca863-b42a-4f84-9d67-b36634e0d9b4\") " Nov 03 22:04:39.538421 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:39.536056 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/933ca863-b42a-4f84-9d67-b36634e0d9b4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/933ca863-b42a-4f84-9d67-b36634e0d9b4-openshift-service-ca\") pod \"933ca863-b42a-4f84-9d67-b36634e0d9b4\" (UID: \"933ca863-b42a-4f84-9d67-b36634e0d9b4\") " Nov 03 22:04:39.538421 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:39.536185 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/933ca863-b42a-4f84-9d67-b36634e0d9b4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/933ca863-b42a-4f84-9d67-b36634e0d9b4-maas-api-tls\") pod \"933ca863-b42a-4f84-9d67-b36634e0d9b4\" (UID: \"933ca863-b42a-4f84-9d67-b36634e0d9b4\") " Nov 03 22:04:39.538421 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:39.536257 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/933ca863-b42a-4f84-9d67-b36634e0d9b4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/933ca863-b42a-4f84-9d67-b36634e0d9b4-metrics-certs\") pod \"933ca863-b42a-4f84-9d67-b36634e0d9b4\" (UID: \"933ca863-b42a-4f84-9d67-b36634e0d9b4\") " Nov 03 22:04:39.538787 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:39.538557 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/933ca863-b42a-4f84-9d67-b36634e0d9b4-openshift-service-ca" pod "933ca863-b42a-4f84-9d67-b36634e0d9b4" (UID: "933ca863-b42a-4f84-9d67-b36634e0d9b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:04:39.544202 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:39.544140 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/933ca863-b42a-4f84-9d67-b36634e0d9b4-maas-api-tls" pod "933ca863-b42a-4f84-9d67-b36634e0d9b4" (UID: "933ca863-b42a-4f84-9d67-b36634e0d9b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:04:39.565760 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:39.565296 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/933ca863-b42a-4f84-9d67-b36634e0d9b4-kube-api-access-j4dvt" pod "933ca863-b42a-4f84-9d67-b36634e0d9b4" (UID: "933ca863-b42a-4f84-9d67-b36634e0d9b4"). InnerVolumeSpecName "kube-api-access-j4dvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:04:39.566320 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:39.566268 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/933ca863-b42a-4f84-9d67-b36634e0d9b4-metrics-certs" pod "933ca863-b42a-4f84-9d67-b36634e0d9b4" (UID: "933ca863-b42a-4f84-9d67-b36634e0d9b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:04:39.638209 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:39.638170 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/933ca863-b42a-4f84-9d67-b36634e0d9b4-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:39.638209 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:39.638202 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/933ca863-b42a-4f84-9d67-b36634e0d9b4-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:39.638209 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:39.638212 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4dvt\" (UniqueName: \"kubernetes.io/projected/933ca863-b42a-4f84-9d67-b36634e0d9b4-kube-api-access-j4dvt\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:39.638462 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:39.638221 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/933ca863-b42a-4f84-9d67-b36634e0d9b4-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:39.781022 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:39.780985 5615 generic.go:356] "Generic (PLEG): container finished" podID="933ca863-b42a-4f84-9d67-b36634e0d9b4" containerID="dff160c8f699bd8be9082207033d5790be32035bd1d808d3d698cbd0ac178842" exitCode=0 Nov 03 22:04:39.781022 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:39.781027 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2" event={"ID":"933ca863-b42a-4f84-9d67-b36634e0d9b4","Type":"ContainerDied","Data":"dff160c8f699bd8be9082207033d5790be32035bd1d808d3d698cbd0ac178842"} Nov 03 22:04:39.781282 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:39.781032 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2" Nov 03 22:04:39.781282 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:39.781090 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2" event={"ID":"933ca863-b42a-4f84-9d67-b36634e0d9b4","Type":"ContainerDied","Data":"b42bfed1ad9d12a7f029323e2a176cdcefb663c47103e9ce9b33c551925502cd"} Nov 03 22:04:39.781282 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:39.781116 5615 scope.go:122] "RemoveContainer" containerID="dff160c8f699bd8be9082207033d5790be32035bd1d808d3d698cbd0ac178842" Nov 03 22:04:39.790927 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:39.790908 5615 scope.go:122] "RemoveContainer" containerID="dff160c8f699bd8be9082207033d5790be32035bd1d808d3d698cbd0ac178842" Nov 03 22:04:39.791298 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:04:39.791273 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dff160c8f699bd8be9082207033d5790be32035bd1d808d3d698cbd0ac178842\": container with ID starting with dff160c8f699bd8be9082207033d5790be32035bd1d808d3d698cbd0ac178842 not found: ID does not exist" containerID="dff160c8f699bd8be9082207033d5790be32035bd1d808d3d698cbd0ac178842" Nov 03 22:04:39.791363 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:39.791304 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff160c8f699bd8be9082207033d5790be32035bd1d808d3d698cbd0ac178842"} err="failed to get container status \"dff160c8f699bd8be9082207033d5790be32035bd1d808d3d698cbd0ac178842\": rpc error: code = NotFound desc = could not find container \"dff160c8f699bd8be9082207033d5790be32035bd1d808d3d698cbd0ac178842\": container with ID starting with dff160c8f699bd8be9082207033d5790be32035bd1d808d3d698cbd0ac178842 not found: ID does not exist" Nov 03 22:04:39.956289 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:39.956203 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2"] Nov 03 22:04:39.963644 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:39.963613 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-320f60-55b647cbd9-qswm2"] Nov 03 22:04:40.621048 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:04:40.620948 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:04:41.652486 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:04:41.652451 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="933ca863-b42a-4f84-9d67-b36634e0d9b4" path="/var/lib/kubelet/pods/933ca863-b42a-4f84-9d67-b36634e0d9b4/volumes" Nov 03 22:05:03.488002 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:03.487960 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp"] Nov 03 22:05:03.488477 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:03.488223 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp" podUID="4271257f-77da-4bd1-9717-f7287bf6b713" containerName="maas-api" containerID="cri-o://b9d1d972590ab62d8739b3f193bf970fe7520dc4c7e69055f37252e1aeaf745f" gracePeriod=30 Nov 03 22:05:03.794360 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:03.794332 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp" Nov 03 22:05:03.846014 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:03.845980 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4271257f-77da-4bd1-9717-f7287bf6b713-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4271257f-77da-4bd1-9717-f7287bf6b713-openshift-service-ca\") pod \"4271257f-77da-4bd1-9717-f7287bf6b713\" (UID: \"4271257f-77da-4bd1-9717-f7287bf6b713\") " Nov 03 22:05:03.846237 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:03.846085 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4271257f-77da-4bd1-9717-f7287bf6b713-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4271257f-77da-4bd1-9717-f7287bf6b713-metrics-certs\") pod \"4271257f-77da-4bd1-9717-f7287bf6b713\" (UID: \"4271257f-77da-4bd1-9717-f7287bf6b713\") " Nov 03 22:05:03.846237 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:03.846211 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4271257f-77da-4bd1-9717-f7287bf6b713-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4271257f-77da-4bd1-9717-f7287bf6b713-maas-api-tls\") pod \"4271257f-77da-4bd1-9717-f7287bf6b713\" (UID: \"4271257f-77da-4bd1-9717-f7287bf6b713\") " Nov 03 22:05:03.846381 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:03.846279 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4271257f-77da-4bd1-9717-f7287bf6b713-kube-api-access-pqrwz\" (UniqueName: \"kubernetes.io/projected/4271257f-77da-4bd1-9717-f7287bf6b713-kube-api-access-pqrwz\") pod \"4271257f-77da-4bd1-9717-f7287bf6b713\" (UID: \"4271257f-77da-4bd1-9717-f7287bf6b713\") " Nov 03 22:05:03.846541 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:03.846469 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4271257f-77da-4bd1-9717-f7287bf6b713-openshift-service-ca" pod "4271257f-77da-4bd1-9717-f7287bf6b713" (UID: "4271257f-77da-4bd1-9717-f7287bf6b713"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:05:03.846862 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:03.846833 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4271257f-77da-4bd1-9717-f7287bf6b713-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:03.848727 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:03.848697 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4271257f-77da-4bd1-9717-f7287bf6b713-kube-api-access-pqrwz" pod "4271257f-77da-4bd1-9717-f7287bf6b713" (UID: "4271257f-77da-4bd1-9717-f7287bf6b713"). InnerVolumeSpecName "kube-api-access-pqrwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:05:03.848727 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:03.848720 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4271257f-77da-4bd1-9717-f7287bf6b713-metrics-certs" pod "4271257f-77da-4bd1-9717-f7287bf6b713" (UID: "4271257f-77da-4bd1-9717-f7287bf6b713"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:05:03.848925 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:03.848816 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4271257f-77da-4bd1-9717-f7287bf6b713-maas-api-tls" pod "4271257f-77da-4bd1-9717-f7287bf6b713" (UID: "4271257f-77da-4bd1-9717-f7287bf6b713"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:05:03.931516 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:03.931479 5615 generic.go:356] "Generic (PLEG): container finished" podID="4271257f-77da-4bd1-9717-f7287bf6b713" containerID="b9d1d972590ab62d8739b3f193bf970fe7520dc4c7e69055f37252e1aeaf745f" exitCode=0 Nov 03 22:05:03.931703 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:03.931547 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp" Nov 03 22:05:03.931703 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:03.931574 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp" event={"ID":"4271257f-77da-4bd1-9717-f7287bf6b713","Type":"ContainerDied","Data":"b9d1d972590ab62d8739b3f193bf970fe7520dc4c7e69055f37252e1aeaf745f"} Nov 03 22:05:03.931703 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:03.931640 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp" event={"ID":"4271257f-77da-4bd1-9717-f7287bf6b713","Type":"ContainerDied","Data":"ea6d4787616d82dee9dbcf174a8dc3564e4640bb29fdcbe939926f9ce1e2ab6f"} Nov 03 22:05:03.931703 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:03.931675 5615 scope.go:122] "RemoveContainer" containerID="b9d1d972590ab62d8739b3f193bf970fe7520dc4c7e69055f37252e1aeaf745f" Nov 03 22:05:03.942454 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:03.942433 5615 scope.go:122] "RemoveContainer" containerID="b9d1d972590ab62d8739b3f193bf970fe7520dc4c7e69055f37252e1aeaf745f" Nov 03 22:05:03.942893 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:05:03.942841 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9d1d972590ab62d8739b3f193bf970fe7520dc4c7e69055f37252e1aeaf745f\": container with ID starting with b9d1d972590ab62d8739b3f193bf970fe7520dc4c7e69055f37252e1aeaf745f not found: ID does not exist" containerID="b9d1d972590ab62d8739b3f193bf970fe7520dc4c7e69055f37252e1aeaf745f" Nov 03 22:05:03.943030 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:03.942901 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9d1d972590ab62d8739b3f193bf970fe7520dc4c7e69055f37252e1aeaf745f"} err="failed to get container status \"b9d1d972590ab62d8739b3f193bf970fe7520dc4c7e69055f37252e1aeaf745f\": rpc error: code = NotFound desc = could not find container \"b9d1d972590ab62d8739b3f193bf970fe7520dc4c7e69055f37252e1aeaf745f\": container with ID starting with b9d1d972590ab62d8739b3f193bf970fe7520dc4c7e69055f37252e1aeaf745f not found: ID does not exist" Nov 03 22:05:03.947774 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:03.947747 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4271257f-77da-4bd1-9717-f7287bf6b713-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:03.947774 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:03.947772 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqrwz\" (UniqueName: \"kubernetes.io/projected/4271257f-77da-4bd1-9717-f7287bf6b713-kube-api-access-pqrwz\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:03.947774 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:03.947782 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4271257f-77da-4bd1-9717-f7287bf6b713-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:04.105327 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:04.105248 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp"] Nov 03 22:05:04.109109 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:04.109064 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0dc66-869c4455b-7bsrp"] Nov 03 22:05:05.651882 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:05.651808 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4271257f-77da-4bd1-9717-f7287bf6b713" path="/var/lib/kubelet/pods/4271257f-77da-4bd1-9717-f7287bf6b713/volumes" Nov 03 22:05:10.619707 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:05:10.619669 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:05:15.765882 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:15.765713 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7"] Nov 03 22:05:30.788386 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:30.788335 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" podUID="232c2524-2557-47c5-9d80-f6235972ca83" containerName="main" containerID="cri-o://891b1f62cb98d0af94477919b238dd7aac6037692825823816da649f63ddc201" gracePeriod=45 Nov 03 22:05:31.054850 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:31.054787 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" Nov 03 22:05:31.127163 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:31.127130 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/232c2524-2557-47c5-9d80-f6235972ca83-kube-api-access-ccpm5\" (UniqueName: \"kubernetes.io/projected/232c2524-2557-47c5-9d80-f6235972ca83-kube-api-access-ccpm5\") pod \"232c2524-2557-47c5-9d80-f6235972ca83\" (UID: \"232c2524-2557-47c5-9d80-f6235972ca83\") " Nov 03 22:05:31.127163 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:31.127174 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/232c2524-2557-47c5-9d80-f6235972ca83-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/232c2524-2557-47c5-9d80-f6235972ca83-tmp-dir\") pod \"232c2524-2557-47c5-9d80-f6235972ca83\" (UID: \"232c2524-2557-47c5-9d80-f6235972ca83\") " Nov 03 22:05:31.127416 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:31.127233 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/232c2524-2557-47c5-9d80-f6235972ca83-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/232c2524-2557-47c5-9d80-f6235972ca83-model-cache\") pod \"232c2524-2557-47c5-9d80-f6235972ca83\" (UID: \"232c2524-2557-47c5-9d80-f6235972ca83\") " Nov 03 22:05:31.127416 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:31.127270 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/232c2524-2557-47c5-9d80-f6235972ca83-tls-certs\" (UniqueName: \"kubernetes.io/secret/232c2524-2557-47c5-9d80-f6235972ca83-tls-certs\") pod \"232c2524-2557-47c5-9d80-f6235972ca83\" (UID: \"232c2524-2557-47c5-9d80-f6235972ca83\") " Nov 03 22:05:31.127416 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:31.127368 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/232c2524-2557-47c5-9d80-f6235972ca83-dshm\" (UniqueName: \"kubernetes.io/empty-dir/232c2524-2557-47c5-9d80-f6235972ca83-dshm\") pod \"232c2524-2557-47c5-9d80-f6235972ca83\" (UID: \"232c2524-2557-47c5-9d80-f6235972ca83\") " Nov 03 22:05:31.127608 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:31.127422 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/232c2524-2557-47c5-9d80-f6235972ca83-home\" (UniqueName: \"kubernetes.io/empty-dir/232c2524-2557-47c5-9d80-f6235972ca83-home\") pod \"232c2524-2557-47c5-9d80-f6235972ca83\" (UID: \"232c2524-2557-47c5-9d80-f6235972ca83\") " Nov 03 22:05:31.127608 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:31.127513 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232c2524-2557-47c5-9d80-f6235972ca83-model-cache" pod "232c2524-2557-47c5-9d80-f6235972ca83" (UID: "232c2524-2557-47c5-9d80-f6235972ca83"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:05:31.127608 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:31.127523 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232c2524-2557-47c5-9d80-f6235972ca83-tmp-dir" pod "232c2524-2557-47c5-9d80-f6235972ca83" (UID: "232c2524-2557-47c5-9d80-f6235972ca83"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:05:31.127915 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:31.127875 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232c2524-2557-47c5-9d80-f6235972ca83-home" pod "232c2524-2557-47c5-9d80-f6235972ca83" (UID: "232c2524-2557-47c5-9d80-f6235972ca83"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:05:31.127915 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:31.127890 5615 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/232c2524-2557-47c5-9d80-f6235972ca83-tmp-dir\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:31.127915 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:31.127910 5615 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/232c2524-2557-47c5-9d80-f6235972ca83-model-cache\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:31.130749 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:31.130724 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/232c2524-2557-47c5-9d80-f6235972ca83-tls-certs" pod "232c2524-2557-47c5-9d80-f6235972ca83" (UID: "232c2524-2557-47c5-9d80-f6235972ca83"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:05:31.130869 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:31.130745 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/232c2524-2557-47c5-9d80-f6235972ca83-kube-api-access-ccpm5" pod "232c2524-2557-47c5-9d80-f6235972ca83" (UID: "232c2524-2557-47c5-9d80-f6235972ca83"). InnerVolumeSpecName "kube-api-access-ccpm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:05:31.130869 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:31.130789 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232c2524-2557-47c5-9d80-f6235972ca83-dshm" pod "232c2524-2557-47c5-9d80-f6235972ca83" (UID: "232c2524-2557-47c5-9d80-f6235972ca83"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:05:31.134380 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:31.134352 5615 generic.go:356] "Generic (PLEG): container finished" podID="232c2524-2557-47c5-9d80-f6235972ca83" containerID="891b1f62cb98d0af94477919b238dd7aac6037692825823816da649f63ddc201" exitCode=0 Nov 03 22:05:31.134511 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:31.134402 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" Nov 03 22:05:31.134511 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:31.134434 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" event={"ID":"232c2524-2557-47c5-9d80-f6235972ca83","Type":"ContainerDied","Data":"891b1f62cb98d0af94477919b238dd7aac6037692825823816da649f63ddc201"} Nov 03 22:05:31.134511 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:31.134476 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7" event={"ID":"232c2524-2557-47c5-9d80-f6235972ca83","Type":"ContainerDied","Data":"ac222e3258aaea13cc488e13570f9e444aaf48fb7e2bfd531300628dee66f426"} Nov 03 22:05:31.134511 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:31.134496 5615 scope.go:122] "RemoveContainer" containerID="891b1f62cb98d0af94477919b238dd7aac6037692825823816da649f63ddc201" Nov 03 22:05:31.155321 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:31.155296 5615 scope.go:122] "RemoveContainer" containerID="891b1f62cb98d0af94477919b238dd7aac6037692825823816da649f63ddc201" Nov 03 22:05:31.155689 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:05:31.155669 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"891b1f62cb98d0af94477919b238dd7aac6037692825823816da649f63ddc201\": container with ID starting with 891b1f62cb98d0af94477919b238dd7aac6037692825823816da649f63ddc201 not found: ID does not exist" containerID="891b1f62cb98d0af94477919b238dd7aac6037692825823816da649f63ddc201" Nov 03 22:05:31.155739 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:31.155703 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"891b1f62cb98d0af94477919b238dd7aac6037692825823816da649f63ddc201"} err="failed to get container status \"891b1f62cb98d0af94477919b238dd7aac6037692825823816da649f63ddc201\": rpc error: code = NotFound desc = could not find container \"891b1f62cb98d0af94477919b238dd7aac6037692825823816da649f63ddc201\": container with ID starting with 891b1f62cb98d0af94477919b238dd7aac6037692825823816da649f63ddc201 not found: ID does not exist" Nov 03 22:05:31.229144 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:31.229103 5615 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/232c2524-2557-47c5-9d80-f6235972ca83-dshm\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:31.229144 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:31.229139 5615 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/232c2524-2557-47c5-9d80-f6235972ca83-home\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:31.229144 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:31.229152 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccpm5\" (UniqueName: \"kubernetes.io/projected/232c2524-2557-47c5-9d80-f6235972ca83-kube-api-access-ccpm5\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:31.229389 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:31.229165 5615 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/232c2524-2557-47c5-9d80-f6235972ca83-tls-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:31.298213 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:31.298121 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7"] Nov 03 22:05:31.310146 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:31.310034 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-791c32/distinct-2-w0-791c32-kserve-69f8d5c46c-2mxf7"] Nov 03 22:05:31.652059 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:31.651984 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="232c2524-2557-47c5-9d80-f6235972ca83" path="/var/lib/kubelet/pods/232c2524-2557-47c5-9d80-f6235972ca83/volumes" Nov 03 22:05:38.239232 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:38.239189 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx"] Nov 03 22:05:40.619292 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:05:40.619247 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:05:53.259673 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:53.259612 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" podUID="473b4351-c417-480c-9974-2f686f617433" containerName="main" containerID="cri-o://9cf35f17d8a59750de34d2ed9a63a13eab662b7d4268973e7fc0188e96832c3b" gracePeriod=45 Nov 03 22:05:53.562626 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:53.562271 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" Nov 03 22:05:53.718960 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:53.718923 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/473b4351-c417-480c-9974-2f686f617433-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/473b4351-c417-480c-9974-2f686f617433-tmp-dir\") pod \"473b4351-c417-480c-9974-2f686f617433\" (UID: \"473b4351-c417-480c-9974-2f686f617433\") " Nov 03 22:05:53.719165 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:53.718989 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/473b4351-c417-480c-9974-2f686f617433-kube-api-access-jfwmf\" (UniqueName: \"kubernetes.io/projected/473b4351-c417-480c-9974-2f686f617433-kube-api-access-jfwmf\") pod \"473b4351-c417-480c-9974-2f686f617433\" (UID: \"473b4351-c417-480c-9974-2f686f617433\") " Nov 03 22:05:53.719165 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:53.719027 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/473b4351-c417-480c-9974-2f686f617433-dshm\" (UniqueName: \"kubernetes.io/empty-dir/473b4351-c417-480c-9974-2f686f617433-dshm\") pod \"473b4351-c417-480c-9974-2f686f617433\" (UID: \"473b4351-c417-480c-9974-2f686f617433\") " Nov 03 22:05:53.719165 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:53.719058 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/473b4351-c417-480c-9974-2f686f617433-tls-certs\" (UniqueName: \"kubernetes.io/secret/473b4351-c417-480c-9974-2f686f617433-tls-certs\") pod \"473b4351-c417-480c-9974-2f686f617433\" (UID: \"473b4351-c417-480c-9974-2f686f617433\") " Nov 03 22:05:53.719354 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:53.719178 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/473b4351-c417-480c-9974-2f686f617433-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/473b4351-c417-480c-9974-2f686f617433-model-cache\") pod \"473b4351-c417-480c-9974-2f686f617433\" (UID: \"473b4351-c417-480c-9974-2f686f617433\") " Nov 03 22:05:53.719354 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:53.719319 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/473b4351-c417-480c-9974-2f686f617433-tmp-dir" pod "473b4351-c417-480c-9974-2f686f617433" (UID: "473b4351-c417-480c-9974-2f686f617433"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:05:53.719473 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:53.719372 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/473b4351-c417-480c-9974-2f686f617433-home\" (UniqueName: \"kubernetes.io/empty-dir/473b4351-c417-480c-9974-2f686f617433-home\") pod \"473b4351-c417-480c-9974-2f686f617433\" (UID: \"473b4351-c417-480c-9974-2f686f617433\") " Nov 03 22:05:53.719538 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:53.719512 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/473b4351-c417-480c-9974-2f686f617433-model-cache" pod "473b4351-c417-480c-9974-2f686f617433" (UID: "473b4351-c417-480c-9974-2f686f617433"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:05:53.719623 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:53.719591 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/473b4351-c417-480c-9974-2f686f617433-home" pod "473b4351-c417-480c-9974-2f686f617433" (UID: "473b4351-c417-480c-9974-2f686f617433"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:05:53.720236 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:53.720215 5615 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/473b4351-c417-480c-9974-2f686f617433-home\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:53.720236 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:53.720237 5615 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/473b4351-c417-480c-9974-2f686f617433-tmp-dir\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:53.720383 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:53.720250 5615 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/473b4351-c417-480c-9974-2f686f617433-model-cache\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:53.721526 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:53.721504 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/473b4351-c417-480c-9974-2f686f617433-dshm" pod "473b4351-c417-480c-9974-2f686f617433" (UID: "473b4351-c417-480c-9974-2f686f617433"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:05:53.721632 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:53.721565 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/473b4351-c417-480c-9974-2f686f617433-tls-certs" pod "473b4351-c417-480c-9974-2f686f617433" (UID: "473b4351-c417-480c-9974-2f686f617433"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:05:53.721632 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:53.721577 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/473b4351-c417-480c-9974-2f686f617433-kube-api-access-jfwmf" pod "473b4351-c417-480c-9974-2f686f617433" (UID: "473b4351-c417-480c-9974-2f686f617433"). InnerVolumeSpecName "kube-api-access-jfwmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:05:53.821255 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:53.821162 5615 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/473b4351-c417-480c-9974-2f686f617433-dshm\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:53.821255 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:53.821196 5615 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/473b4351-c417-480c-9974-2f686f617433-tls-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:53.821255 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:53.821209 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfwmf\" (UniqueName: \"kubernetes.io/projected/473b4351-c417-480c-9974-2f686f617433-kube-api-access-jfwmf\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:54.284125 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:54.284044 5615 generic.go:356] "Generic (PLEG): container finished" podID="473b4351-c417-480c-9974-2f686f617433" containerID="9cf35f17d8a59750de34d2ed9a63a13eab662b7d4268973e7fc0188e96832c3b" exitCode=0 Nov 03 22:05:54.284125 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:54.284100 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" Nov 03 22:05:54.284691 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:54.284132 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" event={"ID":"473b4351-c417-480c-9974-2f686f617433","Type":"ContainerDied","Data":"9cf35f17d8a59750de34d2ed9a63a13eab662b7d4268973e7fc0188e96832c3b"} Nov 03 22:05:54.284691 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:54.284179 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx" event={"ID":"473b4351-c417-480c-9974-2f686f617433","Type":"ContainerDied","Data":"592f1dd53f8ab3c0c2f759823e4e8fd932d5453ca807fd26decbadbc2bb8aaa7"} Nov 03 22:05:54.284691 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:54.284202 5615 scope.go:122] "RemoveContainer" containerID="9cf35f17d8a59750de34d2ed9a63a13eab662b7d4268973e7fc0188e96832c3b" Nov 03 22:05:54.297281 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:54.296897 5615 scope.go:122] "RemoveContainer" containerID="9cf35f17d8a59750de34d2ed9a63a13eab662b7d4268973e7fc0188e96832c3b" Nov 03 22:05:54.297580 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:05:54.297540 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cf35f17d8a59750de34d2ed9a63a13eab662b7d4268973e7fc0188e96832c3b\": container with ID starting with 9cf35f17d8a59750de34d2ed9a63a13eab662b7d4268973e7fc0188e96832c3b not found: ID does not exist" containerID="9cf35f17d8a59750de34d2ed9a63a13eab662b7d4268973e7fc0188e96832c3b" Nov 03 22:05:54.297655 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:54.297593 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cf35f17d8a59750de34d2ed9a63a13eab662b7d4268973e7fc0188e96832c3b"} err="failed to get container status \"9cf35f17d8a59750de34d2ed9a63a13eab662b7d4268973e7fc0188e96832c3b\": rpc error: code = NotFound desc = could not find container \"9cf35f17d8a59750de34d2ed9a63a13eab662b7d4268973e7fc0188e96832c3b\": container with ID starting with 9cf35f17d8a59750de34d2ed9a63a13eab662b7d4268973e7fc0188e96832c3b not found: ID does not exist" Nov 03 22:05:54.451543 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:54.451501 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx"] Nov 03 22:05:54.455420 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:54.455380 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-4d1799/rate-test-model-4d1799-kserve-58fd9cb4df-lxwfx"] Nov 03 22:05:55.613725 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:55.613683 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr"] Nov 03 22:05:55.614212 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:55.613953 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr" podUID="ce51a38f-47a9-4926-adec-cecfceba6000" containerName="maas-api" containerID="cri-o://831ac3c020c9166e642c67066767332bd165c1f91e62b25a0231bde74d233927" gracePeriod=30 Nov 03 22:05:55.652406 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:55.652365 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="473b4351-c417-480c-9974-2f686f617433" path="/var/lib/kubelet/pods/473b4351-c417-480c-9974-2f686f617433/volumes" Nov 03 22:05:55.902408 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:55.902378 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr" Nov 03 22:05:56.056705 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:56.056671 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce51a38f-47a9-4926-adec-cecfceba6000-kube-api-access-gfhxk\" (UniqueName: \"kubernetes.io/projected/ce51a38f-47a9-4926-adec-cecfceba6000-kube-api-access-gfhxk\") pod \"ce51a38f-47a9-4926-adec-cecfceba6000\" (UID: \"ce51a38f-47a9-4926-adec-cecfceba6000\") " Nov 03 22:05:56.056908 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:56.056730 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce51a38f-47a9-4926-adec-cecfceba6000-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce51a38f-47a9-4926-adec-cecfceba6000-metrics-certs\") pod \"ce51a38f-47a9-4926-adec-cecfceba6000\" (UID: \"ce51a38f-47a9-4926-adec-cecfceba6000\") " Nov 03 22:05:56.056908 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:56.056860 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce51a38f-47a9-4926-adec-cecfceba6000-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce51a38f-47a9-4926-adec-cecfceba6000-maas-api-tls\") pod \"ce51a38f-47a9-4926-adec-cecfceba6000\" (UID: \"ce51a38f-47a9-4926-adec-cecfceba6000\") " Nov 03 22:05:56.056908 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:56.056900 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce51a38f-47a9-4926-adec-cecfceba6000-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce51a38f-47a9-4926-adec-cecfceba6000-openshift-service-ca\") pod \"ce51a38f-47a9-4926-adec-cecfceba6000\" (UID: \"ce51a38f-47a9-4926-adec-cecfceba6000\") " Nov 03 22:05:56.057597 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:56.057545 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce51a38f-47a9-4926-adec-cecfceba6000-openshift-service-ca" pod "ce51a38f-47a9-4926-adec-cecfceba6000" (UID: "ce51a38f-47a9-4926-adec-cecfceba6000"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:05:56.059495 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:56.059460 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce51a38f-47a9-4926-adec-cecfceba6000-maas-api-tls" pod "ce51a38f-47a9-4926-adec-cecfceba6000" (UID: "ce51a38f-47a9-4926-adec-cecfceba6000"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:05:56.059629 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:56.059550 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce51a38f-47a9-4926-adec-cecfceba6000-kube-api-access-gfhxk" pod "ce51a38f-47a9-4926-adec-cecfceba6000" (UID: "ce51a38f-47a9-4926-adec-cecfceba6000"). InnerVolumeSpecName "kube-api-access-gfhxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:05:56.059629 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:56.059575 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce51a38f-47a9-4926-adec-cecfceba6000-metrics-certs" pod "ce51a38f-47a9-4926-adec-cecfceba6000" (UID: "ce51a38f-47a9-4926-adec-cecfceba6000"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:05:56.159613 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:56.159421 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce51a38f-47a9-4926-adec-cecfceba6000-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:56.159613 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:56.159469 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce51a38f-47a9-4926-adec-cecfceba6000-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:56.159613 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:56.159485 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfhxk\" (UniqueName: \"kubernetes.io/projected/ce51a38f-47a9-4926-adec-cecfceba6000-kube-api-access-gfhxk\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:56.159613 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:56.159500 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce51a38f-47a9-4926-adec-cecfceba6000-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:56.298998 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:56.298963 5615 generic.go:356] "Generic (PLEG): container finished" podID="ce51a38f-47a9-4926-adec-cecfceba6000" containerID="831ac3c020c9166e642c67066767332bd165c1f91e62b25a0231bde74d233927" exitCode=0 Nov 03 22:05:56.299225 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:56.299013 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr" Nov 03 22:05:56.299225 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:56.299053 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr" event={"ID":"ce51a38f-47a9-4926-adec-cecfceba6000","Type":"ContainerDied","Data":"831ac3c020c9166e642c67066767332bd165c1f91e62b25a0231bde74d233927"} Nov 03 22:05:56.299225 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:56.299129 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr" event={"ID":"ce51a38f-47a9-4926-adec-cecfceba6000","Type":"ContainerDied","Data":"c3c0ed092d705203ba81286266ab4599dd561db7c705146892d5b7d4f4ad99f6"} Nov 03 22:05:56.299225 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:56.299160 5615 scope.go:122] "RemoveContainer" containerID="831ac3c020c9166e642c67066767332bd165c1f91e62b25a0231bde74d233927" Nov 03 22:05:56.309980 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:56.309959 5615 scope.go:122] "RemoveContainer" containerID="831ac3c020c9166e642c67066767332bd165c1f91e62b25a0231bde74d233927" Nov 03 22:05:56.310289 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:05:56.310267 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"831ac3c020c9166e642c67066767332bd165c1f91e62b25a0231bde74d233927\": container with ID starting with 831ac3c020c9166e642c67066767332bd165c1f91e62b25a0231bde74d233927 not found: ID does not exist" containerID="831ac3c020c9166e642c67066767332bd165c1f91e62b25a0231bde74d233927" Nov 03 22:05:56.310353 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:56.310299 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"831ac3c020c9166e642c67066767332bd165c1f91e62b25a0231bde74d233927"} err="failed to get container status \"831ac3c020c9166e642c67066767332bd165c1f91e62b25a0231bde74d233927\": rpc error: code = NotFound desc = could not find container \"831ac3c020c9166e642c67066767332bd165c1f91e62b25a0231bde74d233927\": container with ID starting with 831ac3c020c9166e642c67066767332bd165c1f91e62b25a0231bde74d233927 not found: ID does not exist" Nov 03 22:05:56.466855 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:56.466767 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr"] Nov 03 22:05:56.470767 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:56.470740 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-791c32-86fdfcb58-fmfhr"] Nov 03 22:05:57.651722 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:05:57.651680 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce51a38f-47a9-4926-adec-cecfceba6000" path="/var/lib/kubelet/pods/ce51a38f-47a9-4926-adec-cecfceba6000/volumes" Nov 03 22:06:10.619143 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:06:10.619065 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:06:29.588724 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:29.588687 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7"] Nov 03 22:06:29.589196 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:29.588919 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7" podUID="dca66826-bea7-4c51-8fc0-b0e824ac5f2c" containerName="maas-api" containerID="cri-o://f98b553f1c4b219392f6f24a0d21c6876f5cb0cafd6ed433a90eb56426463e89" gracePeriod=30 Nov 03 22:06:29.891770 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:29.891739 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7" Nov 03 22:06:29.966429 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:29.966392 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dca66826-bea7-4c51-8fc0-b0e824ac5f2c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca66826-bea7-4c51-8fc0-b0e824ac5f2c-maas-api-tls\") pod \"dca66826-bea7-4c51-8fc0-b0e824ac5f2c\" (UID: \"dca66826-bea7-4c51-8fc0-b0e824ac5f2c\") " Nov 03 22:06:29.978953 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:29.978918 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dca66826-bea7-4c51-8fc0-b0e824ac5f2c-kube-api-access-njg54\" (UniqueName: \"kubernetes.io/projected/dca66826-bea7-4c51-8fc0-b0e824ac5f2c-kube-api-access-njg54\") pod \"dca66826-bea7-4c51-8fc0-b0e824ac5f2c\" (UID: \"dca66826-bea7-4c51-8fc0-b0e824ac5f2c\") " Nov 03 22:06:29.979323 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:29.979306 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dca66826-bea7-4c51-8fc0-b0e824ac5f2c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca66826-bea7-4c51-8fc0-b0e824ac5f2c-openshift-service-ca\") pod \"dca66826-bea7-4c51-8fc0-b0e824ac5f2c\" (UID: \"dca66826-bea7-4c51-8fc0-b0e824ac5f2c\") " Nov 03 22:06:29.979474 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:29.979457 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dca66826-bea7-4c51-8fc0-b0e824ac5f2c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca66826-bea7-4c51-8fc0-b0e824ac5f2c-metrics-certs\") pod \"dca66826-bea7-4c51-8fc0-b0e824ac5f2c\" (UID: \"dca66826-bea7-4c51-8fc0-b0e824ac5f2c\") " Nov 03 22:06:29.979975 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:29.979943 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dca66826-bea7-4c51-8fc0-b0e824ac5f2c-openshift-service-ca" pod "dca66826-bea7-4c51-8fc0-b0e824ac5f2c" (UID: "dca66826-bea7-4c51-8fc0-b0e824ac5f2c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:06:29.980163 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:29.980144 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca66826-bea7-4c51-8fc0-b0e824ac5f2c-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:29.989711 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:29.989666 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dca66826-bea7-4c51-8fc0-b0e824ac5f2c-kube-api-access-njg54" pod "dca66826-bea7-4c51-8fc0-b0e824ac5f2c" (UID: "dca66826-bea7-4c51-8fc0-b0e824ac5f2c"). InnerVolumeSpecName "kube-api-access-njg54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:06:29.989884 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:29.989842 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dca66826-bea7-4c51-8fc0-b0e824ac5f2c-maas-api-tls" pod "dca66826-bea7-4c51-8fc0-b0e824ac5f2c" (UID: "dca66826-bea7-4c51-8fc0-b0e824ac5f2c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:06:29.991492 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:29.991460 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dca66826-bea7-4c51-8fc0-b0e824ac5f2c-metrics-certs" pod "dca66826-bea7-4c51-8fc0-b0e824ac5f2c" (UID: "dca66826-bea7-4c51-8fc0-b0e824ac5f2c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:06:30.090117 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:30.090050 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njg54\" (UniqueName: \"kubernetes.io/projected/dca66826-bea7-4c51-8fc0-b0e824ac5f2c-kube-api-access-njg54\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:30.090117 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:30.090118 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca66826-bea7-4c51-8fc0-b0e824ac5f2c-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:30.090401 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:30.090136 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca66826-bea7-4c51-8fc0-b0e824ac5f2c-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:30.556958 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:30.556916 5615 generic.go:356] "Generic (PLEG): container finished" podID="dca66826-bea7-4c51-8fc0-b0e824ac5f2c" containerID="f98b553f1c4b219392f6f24a0d21c6876f5cb0cafd6ed433a90eb56426463e89" exitCode=0 Nov 03 22:06:30.557279 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:30.557250 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7" event={"ID":"dca66826-bea7-4c51-8fc0-b0e824ac5f2c","Type":"ContainerDied","Data":"f98b553f1c4b219392f6f24a0d21c6876f5cb0cafd6ed433a90eb56426463e89"} Nov 03 22:06:30.557481 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:30.557464 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7" event={"ID":"dca66826-bea7-4c51-8fc0-b0e824ac5f2c","Type":"ContainerDied","Data":"08352a8179ff4ad1153c5a4c29bf24164d7415b7891cde6593f70fb1261d9eb6"} Nov 03 22:06:30.557614 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:30.557599 5615 scope.go:122] "RemoveContainer" containerID="f98b553f1c4b219392f6f24a0d21c6876f5cb0cafd6ed433a90eb56426463e89" Nov 03 22:06:30.557940 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:30.557891 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7" Nov 03 22:06:30.577700 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:30.577671 5615 scope.go:122] "RemoveContainer" containerID="f98b553f1c4b219392f6f24a0d21c6876f5cb0cafd6ed433a90eb56426463e89" Nov 03 22:06:30.578184 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:06:30.578151 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f98b553f1c4b219392f6f24a0d21c6876f5cb0cafd6ed433a90eb56426463e89\": container with ID starting with f98b553f1c4b219392f6f24a0d21c6876f5cb0cafd6ed433a90eb56426463e89 not found: ID does not exist" containerID="f98b553f1c4b219392f6f24a0d21c6876f5cb0cafd6ed433a90eb56426463e89" Nov 03 22:06:30.578304 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:30.578197 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f98b553f1c4b219392f6f24a0d21c6876f5cb0cafd6ed433a90eb56426463e89"} err="failed to get container status \"f98b553f1c4b219392f6f24a0d21c6876f5cb0cafd6ed433a90eb56426463e89\": rpc error: code = NotFound desc = could not find container \"f98b553f1c4b219392f6f24a0d21c6876f5cb0cafd6ed433a90eb56426463e89\": container with ID starting with f98b553f1c4b219392f6f24a0d21c6876f5cb0cafd6ed433a90eb56426463e89 not found: ID does not exist" Nov 03 22:06:30.728919 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:30.728885 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7"] Nov 03 22:06:30.733263 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:30.733227 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d1799-99d5fbd7c-mfxz7"] Nov 03 22:06:31.651589 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:31.651552 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dca66826-bea7-4c51-8fc0-b0e824ac5f2c" path="/var/lib/kubelet/pods/dca66826-bea7-4c51-8fc0-b0e824ac5f2c/volumes" Nov 03 22:06:39.657458 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:39.657378 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp"] Nov 03 22:06:39.657903 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:39.657589 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" podUID="ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829" containerName="istio-proxy" containerID="cri-o://6be1ac7260734d89285b74eedb028c32728be6f4bbe585e835863dc4c7d267e3" gracePeriod=30 Nov 03 22:06:40.620247 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:06:40.620209 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:06:43.794225 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:43.794182 5615 patch_prober.go:29] interesting pod/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:06:43.794635 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:43.794252 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" podUID="ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:06:44.923730 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:44.923661 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 22:06:44.972860 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:44.972830 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-workload-certs\") pod \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " Nov 03 22:06:44.973053 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:44.972914 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-kube-api-access-b2d5z\" (UniqueName: \"kubernetes.io/projected/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-kube-api-access-b2d5z\") pod \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " Nov 03 22:06:44.973053 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:44.972941 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istio-podinfo\") pod \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " Nov 03 22:06:44.973053 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:44.972975 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istio-envoy\") pod \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " Nov 03 22:06:44.973215 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:44.973122 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istiod-ca-cert\") pod \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " Nov 03 22:06:44.973215 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:44.973169 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-workload-certs" pod "ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829" (UID: "ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:06:44.973299 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:44.973231 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istio-data\") pod \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " Nov 03 22:06:44.973357 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:44.973312 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istio-token\" (UniqueName: \"kubernetes.io/projected/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istio-token\") pod \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " Nov 03 22:06:44.973357 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:44.973345 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-credential-socket\") pod \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " Nov 03 22:06:44.973471 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:44.973390 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-workload-socket\") pod \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\" (UID: \"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829\") " Nov 03 22:06:44.973631 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:44.973606 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istio-data" pod "ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829" (UID: "ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:06:44.973631 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:44.973620 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istiod-ca-cert" pod "ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829" (UID: "ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:06:44.973870 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:44.973841 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-workload-socket" pod "ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829" (UID: "ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:06:44.974008 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:44.973965 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-credential-socket" pod "ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829" (UID: "ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:06:44.974184 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:44.974163 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:44.974271 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:44.974190 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:44.974271 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:44.974216 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:44.974271 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:44.974232 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:44.974271 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:44.974248 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:44.975563 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:44.975536 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-kube-api-access-b2d5z" pod "ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829" (UID: "ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829"). InnerVolumeSpecName "kube-api-access-b2d5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:06:44.975726 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:44.975708 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istio-token" pod "ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829" (UID: "ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:06:44.975812 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:44.975798 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istio-envoy" pod "ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829" (UID: "ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:06:44.975865 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:44.975844 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istio-podinfo" pod "ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829" (UID: "ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:06:45.076312 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:45.076275 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2d5z\" (UniqueName: \"kubernetes.io/projected/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-kube-api-access-b2d5z\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:45.076312 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:45.076306 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:45.076312 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:45.076317 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:45.076570 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:45.076326 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:45.674584 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:45.674547 5615 generic.go:356] "Generic (PLEG): container finished" podID="ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829" containerID="6be1ac7260734d89285b74eedb028c32728be6f4bbe585e835863dc4c7d267e3" exitCode=0 Nov 03 22:06:45.674744 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:45.674602 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" Nov 03 22:06:45.674744 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:45.674629 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" event={"ID":"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829","Type":"ContainerDied","Data":"6be1ac7260734d89285b74eedb028c32728be6f4bbe585e835863dc4c7d267e3"} Nov 03 22:06:45.674744 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:45.674689 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp" event={"ID":"ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829","Type":"ContainerDied","Data":"720aefc4a14536b3fd9caf952ee42eee41cf31fc51b161073fac367843cd7c51"} Nov 03 22:06:45.674744 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:45.674711 5615 scope.go:122] "RemoveContainer" containerID="6be1ac7260734d89285b74eedb028c32728be6f4bbe585e835863dc4c7d267e3" Nov 03 22:06:45.684773 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:45.684745 5615 scope.go:122] "RemoveContainer" containerID="7e3e4956bfdeeb65e4b0e46439f7a10fde5a99e478ff9f1e4216ff398956183e" Nov 03 22:06:45.693671 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:45.693645 5615 scope.go:122] "RemoveContainer" containerID="6be1ac7260734d89285b74eedb028c32728be6f4bbe585e835863dc4c7d267e3" Nov 03 22:06:45.694008 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:06:45.693987 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6be1ac7260734d89285b74eedb028c32728be6f4bbe585e835863dc4c7d267e3\": container with ID starting with 6be1ac7260734d89285b74eedb028c32728be6f4bbe585e835863dc4c7d267e3 not found: ID does not exist" containerID="6be1ac7260734d89285b74eedb028c32728be6f4bbe585e835863dc4c7d267e3" Nov 03 22:06:45.694067 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:45.694019 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6be1ac7260734d89285b74eedb028c32728be6f4bbe585e835863dc4c7d267e3"} err="failed to get container status \"6be1ac7260734d89285b74eedb028c32728be6f4bbe585e835863dc4c7d267e3\": rpc error: code = NotFound desc = could not find container \"6be1ac7260734d89285b74eedb028c32728be6f4bbe585e835863dc4c7d267e3\": container with ID starting with 6be1ac7260734d89285b74eedb028c32728be6f4bbe585e835863dc4c7d267e3 not found: ID does not exist" Nov 03 22:06:45.694067 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:45.694038 5615 scope.go:122] "RemoveContainer" containerID="7e3e4956bfdeeb65e4b0e46439f7a10fde5a99e478ff9f1e4216ff398956183e" Nov 03 22:06:45.694351 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:06:45.694330 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e3e4956bfdeeb65e4b0e46439f7a10fde5a99e478ff9f1e4216ff398956183e\": container with ID starting with 7e3e4956bfdeeb65e4b0e46439f7a10fde5a99e478ff9f1e4216ff398956183e not found: ID does not exist" containerID="7e3e4956bfdeeb65e4b0e46439f7a10fde5a99e478ff9f1e4216ff398956183e" Nov 03 22:06:45.694408 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:45.694358 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e3e4956bfdeeb65e4b0e46439f7a10fde5a99e478ff9f1e4216ff398956183e"} err="failed to get container status \"7e3e4956bfdeeb65e4b0e46439f7a10fde5a99e478ff9f1e4216ff398956183e\": rpc error: code = NotFound desc = could not find container \"7e3e4956bfdeeb65e4b0e46439f7a10fde5a99e478ff9f1e4216ff398956183e\": container with ID starting with 7e3e4956bfdeeb65e4b0e46439f7a10fde5a99e478ff9f1e4216ff398956183e not found: ID does not exist" Nov 03 22:06:45.896944 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:45.896913 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp"] Nov 03 22:06:45.947928 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:45.947839 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-4d1799-openshift-default-89c45d9d5-bspgp"] Nov 03 22:06:47.651510 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:06:47.651475 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829" path="/var/lib/kubelet/pods/ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829/volumes" Nov 03 22:07:10.618462 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:07:10.618420 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:07:11.942131 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:11.942050 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s"] Nov 03 22:07:12.452462 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:12.452424 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx"] Nov 03 22:07:16.943502 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:16.943447 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s" podUID="cb894762-26e0-4dd6-9e55-0a0aed4455c9" containerName="payload-pre-processing" containerID="cri-o://426167e2eb127abef614e6002ae6b0e53dd17cfd61b7e005ca240e1071d557e0" gracePeriod=25 Nov 03 22:07:21.879886 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:21.879844 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s" podUID="cb894762-26e0-4dd6-9e55-0a0aed4455c9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:07:24.243875 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:24.243832 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt"] Nov 03 22:07:24.244340 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:24.244019 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" podUID="ad38eb17-97aa-4877-9efb-7e324f3ff5bd" containerName="istio-proxy" containerID="cri-o://ce6ea373bc6574b8e309d0e99d458bee391a930bd8951b50a79b709a99b30c66" gracePeriod=30 Nov 03 22:07:27.396430 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:27.396391 5615 patch_prober.go:29] interesting pod/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:07:27.396906 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:27.396464 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" podUID="ad38eb17-97aa-4877-9efb-7e324f3ff5bd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:07:27.473058 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:27.473013 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" podUID="fd34f512-e1e0-433b-997f-b124daa6ded0" containerName="main" containerID="cri-o://e8372699dcdf0287dbaaf6f2451adbba07ee24e494812c888d1d6cc77db9ffa8" gracePeriod=45 Nov 03 22:07:27.726000 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:27.725972 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" Nov 03 22:07:27.878652 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:27.878615 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd34f512-e1e0-433b-997f-b124daa6ded0-kube-api-access-8g2kt\" (UniqueName: \"kubernetes.io/projected/fd34f512-e1e0-433b-997f-b124daa6ded0-kube-api-access-8g2kt\") pod \"fd34f512-e1e0-433b-997f-b124daa6ded0\" (UID: \"fd34f512-e1e0-433b-997f-b124daa6ded0\") " Nov 03 22:07:27.878830 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:27.878685 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd34f512-e1e0-433b-997f-b124daa6ded0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd34f512-e1e0-433b-997f-b124daa6ded0-tmp-dir\") pod \"fd34f512-e1e0-433b-997f-b124daa6ded0\" (UID: \"fd34f512-e1e0-433b-997f-b124daa6ded0\") " Nov 03 22:07:27.878830 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:27.878743 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd34f512-e1e0-433b-997f-b124daa6ded0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd34f512-e1e0-433b-997f-b124daa6ded0-dshm\") pod \"fd34f512-e1e0-433b-997f-b124daa6ded0\" (UID: \"fd34f512-e1e0-433b-997f-b124daa6ded0\") " Nov 03 22:07:27.878916 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:27.878872 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd34f512-e1e0-433b-997f-b124daa6ded0-tls-certs\" (UniqueName: \"kubernetes.io/secret/fd34f512-e1e0-433b-997f-b124daa6ded0-tls-certs\") pod \"fd34f512-e1e0-433b-997f-b124daa6ded0\" (UID: \"fd34f512-e1e0-433b-997f-b124daa6ded0\") " Nov 03 22:07:27.878964 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:27.878916 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd34f512-e1e0-433b-997f-b124daa6ded0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd34f512-e1e0-433b-997f-b124daa6ded0-model-cache\") pod \"fd34f512-e1e0-433b-997f-b124daa6ded0\" (UID: \"fd34f512-e1e0-433b-997f-b124daa6ded0\") " Nov 03 22:07:27.878964 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:27.878946 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd34f512-e1e0-433b-997f-b124daa6ded0-home\" (UniqueName: \"kubernetes.io/empty-dir/fd34f512-e1e0-433b-997f-b124daa6ded0-home\") pod \"fd34f512-e1e0-433b-997f-b124daa6ded0\" (UID: \"fd34f512-e1e0-433b-997f-b124daa6ded0\") " Nov 03 22:07:27.879101 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:27.879036 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd34f512-e1e0-433b-997f-b124daa6ded0-tmp-dir" pod "fd34f512-e1e0-433b-997f-b124daa6ded0" (UID: "fd34f512-e1e0-433b-997f-b124daa6ded0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:27.879254 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:27.879227 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd34f512-e1e0-433b-997f-b124daa6ded0-model-cache" pod "fd34f512-e1e0-433b-997f-b124daa6ded0" (UID: "fd34f512-e1e0-433b-997f-b124daa6ded0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:27.879370 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:27.879348 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd34f512-e1e0-433b-997f-b124daa6ded0-home" pod "fd34f512-e1e0-433b-997f-b124daa6ded0" (UID: "fd34f512-e1e0-433b-997f-b124daa6ded0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:27.879577 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:27.879558 5615 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd34f512-e1e0-433b-997f-b124daa6ded0-tmp-dir\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:27.879661 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:27.879581 5615 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd34f512-e1e0-433b-997f-b124daa6ded0-model-cache\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:27.879661 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:27.879598 5615 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd34f512-e1e0-433b-997f-b124daa6ded0-home\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:27.881512 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:27.881487 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd34f512-e1e0-433b-997f-b124daa6ded0-dshm" pod "fd34f512-e1e0-433b-997f-b124daa6ded0" (UID: "fd34f512-e1e0-433b-997f-b124daa6ded0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:27.881637 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:27.881530 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd34f512-e1e0-433b-997f-b124daa6ded0-kube-api-access-8g2kt" pod "fd34f512-e1e0-433b-997f-b124daa6ded0" (UID: "fd34f512-e1e0-433b-997f-b124daa6ded0"). InnerVolumeSpecName "kube-api-access-8g2kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:07:27.881637 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:27.881616 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd34f512-e1e0-433b-997f-b124daa6ded0-tls-certs" pod "fd34f512-e1e0-433b-997f-b124daa6ded0" (UID: "fd34f512-e1e0-433b-997f-b124daa6ded0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:07:27.911138 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:27.911050 5615 generic.go:356] "Generic (PLEG): container finished" podID="fd34f512-e1e0-433b-997f-b124daa6ded0" containerID="e8372699dcdf0287dbaaf6f2451adbba07ee24e494812c888d1d6cc77db9ffa8" exitCode=0 Nov 03 22:07:27.911138 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:27.911122 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" Nov 03 22:07:27.911317 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:27.911139 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" event={"ID":"fd34f512-e1e0-433b-997f-b124daa6ded0","Type":"ContainerDied","Data":"e8372699dcdf0287dbaaf6f2451adbba07ee24e494812c888d1d6cc77db9ffa8"} Nov 03 22:07:27.911317 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:27.911228 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" event={"ID":"fd34f512-e1e0-433b-997f-b124daa6ded0","Type":"ContainerDied","Data":"46a4aa88f79043a8ae9b980aadd58d726f8030271d2eb2e51e7a41e79b3439e5"} Nov 03 22:07:27.911317 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:27.911260 5615 scope.go:122] "RemoveContainer" containerID="e8372699dcdf0287dbaaf6f2451adbba07ee24e494812c888d1d6cc77db9ffa8" Nov 03 22:07:27.922857 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:27.922832 5615 scope.go:122] "RemoveContainer" containerID="e8372699dcdf0287dbaaf6f2451adbba07ee24e494812c888d1d6cc77db9ffa8" Nov 03 22:07:27.923183 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:07:27.923161 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8372699dcdf0287dbaaf6f2451adbba07ee24e494812c888d1d6cc77db9ffa8\": container with ID starting with e8372699dcdf0287dbaaf6f2451adbba07ee24e494812c888d1d6cc77db9ffa8 not found: ID does not exist" containerID="e8372699dcdf0287dbaaf6f2451adbba07ee24e494812c888d1d6cc77db9ffa8" Nov 03 22:07:27.923286 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:27.923192 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8372699dcdf0287dbaaf6f2451adbba07ee24e494812c888d1d6cc77db9ffa8"} err="failed to get container status \"e8372699dcdf0287dbaaf6f2451adbba07ee24e494812c888d1d6cc77db9ffa8\": rpc error: code = NotFound desc = could not find container \"e8372699dcdf0287dbaaf6f2451adbba07ee24e494812c888d1d6cc77db9ffa8\": container with ID starting with e8372699dcdf0287dbaaf6f2451adbba07ee24e494812c888d1d6cc77db9ffa8 not found: ID does not exist" Nov 03 22:07:27.984115 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:27.983628 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g2kt\" (UniqueName: \"kubernetes.io/projected/fd34f512-e1e0-433b-997f-b124daa6ded0-kube-api-access-8g2kt\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:27.984115 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:27.983670 5615 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd34f512-e1e0-433b-997f-b124daa6ded0-dshm\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:27.984115 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:27.983694 5615 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd34f512-e1e0-433b-997f-b124daa6ded0-tls-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:28.079188 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:28.079154 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx"] Nov 03 22:07:28.083385 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:28.083352 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx"] Nov 03 22:07:28.655174 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:28.655128 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-a6219b/rate-test-model-a6219b-kserve-5678ccf457-xp2xx" podUID="fd34f512-e1e0-433b-997f-b124daa6ded0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.189:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:07:29.504153 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.504125 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 22:07:29.517761 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.517686 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-kube-api-access-c78dk\" (UniqueName: \"kubernetes.io/projected/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-kube-api-access-c78dk\") pod \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " Nov 03 22:07:29.517761 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.517744 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-workload-socket\") pod \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " Nov 03 22:07:29.517966 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.517785 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istio-token\" (UniqueName: \"kubernetes.io/projected/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istio-token\") pod \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " Nov 03 22:07:29.517966 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.517837 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istio-envoy\") pod \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " Nov 03 22:07:29.517966 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.517872 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istio-podinfo\") pod \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " Nov 03 22:07:29.518162 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.517968 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istio-data\") pod \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " Nov 03 22:07:29.518162 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.518016 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-workload-certs\") pod \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " Nov 03 22:07:29.518162 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.518111 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-credential-socket\") pod \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " Nov 03 22:07:29.518162 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.518139 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-workload-socket" pod "ad38eb17-97aa-4877-9efb-7e324f3ff5bd" (UID: "ad38eb17-97aa-4877-9efb-7e324f3ff5bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:29.518393 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.518168 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istiod-ca-cert\") pod \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\" (UID: \"ad38eb17-97aa-4877-9efb-7e324f3ff5bd\") " Nov 03 22:07:29.518393 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.518356 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-workload-certs" pod "ad38eb17-97aa-4877-9efb-7e324f3ff5bd" (UID: "ad38eb17-97aa-4877-9efb-7e324f3ff5bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:29.518663 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.518636 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-credential-socket" pod "ad38eb17-97aa-4877-9efb-7e324f3ff5bd" (UID: "ad38eb17-97aa-4877-9efb-7e324f3ff5bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:29.518771 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.518669 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istio-data" pod "ad38eb17-97aa-4877-9efb-7e324f3ff5bd" (UID: "ad38eb17-97aa-4877-9efb-7e324f3ff5bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:29.518771 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.518729 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:29.518771 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.518749 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:29.518771 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.518748 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istiod-ca-cert" pod "ad38eb17-97aa-4877-9efb-7e324f3ff5bd" (UID: "ad38eb17-97aa-4877-9efb-7e324f3ff5bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:07:29.520360 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.520331 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istio-envoy" pod "ad38eb17-97aa-4877-9efb-7e324f3ff5bd" (UID: "ad38eb17-97aa-4877-9efb-7e324f3ff5bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:29.520536 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.520506 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istio-podinfo" pod "ad38eb17-97aa-4877-9efb-7e324f3ff5bd" (UID: "ad38eb17-97aa-4877-9efb-7e324f3ff5bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:07:29.520641 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.520612 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-kube-api-access-c78dk" pod "ad38eb17-97aa-4877-9efb-7e324f3ff5bd" (UID: "ad38eb17-97aa-4877-9efb-7e324f3ff5bd"). InnerVolumeSpecName "kube-api-access-c78dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:07:29.520694 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.520641 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istio-token" pod "ad38eb17-97aa-4877-9efb-7e324f3ff5bd" (UID: "ad38eb17-97aa-4877-9efb-7e324f3ff5bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:07:29.619690 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.619643 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:29.619690 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.619690 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:29.619956 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.619706 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:29.619956 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.619721 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c78dk\" (UniqueName: \"kubernetes.io/projected/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-kube-api-access-c78dk\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:29.619956 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.619738 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:29.619956 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.619751 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:29.619956 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.619766 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad38eb17-97aa-4877-9efb-7e324f3ff5bd-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:29.652445 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.652406 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd34f512-e1e0-433b-997f-b124daa6ded0" path="/var/lib/kubelet/pods/fd34f512-e1e0-433b-997f-b124daa6ded0/volumes" Nov 03 22:07:29.665924 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.665892 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6"] Nov 03 22:07:29.680034 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.679989 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz"] Nov 03 22:07:29.932414 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.932325 5615 generic.go:356] "Generic (PLEG): container finished" podID="ad38eb17-97aa-4877-9efb-7e324f3ff5bd" containerID="ce6ea373bc6574b8e309d0e99d458bee391a930bd8951b50a79b709a99b30c66" exitCode=0 Nov 03 22:07:29.932414 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.932372 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" event={"ID":"ad38eb17-97aa-4877-9efb-7e324f3ff5bd","Type":"ContainerDied","Data":"ce6ea373bc6574b8e309d0e99d458bee391a930bd8951b50a79b709a99b30c66"} Nov 03 22:07:29.932623 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.932419 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" event={"ID":"ad38eb17-97aa-4877-9efb-7e324f3ff5bd","Type":"ContainerDied","Data":"aec92b4562f0b25c5915b5da6d767a1d14d33293c81ce6a065f5fa1ee1bbfaef"} Nov 03 22:07:29.932623 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.932377 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt" Nov 03 22:07:29.932623 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.932445 5615 scope.go:122] "RemoveContainer" containerID="ce6ea373bc6574b8e309d0e99d458bee391a930bd8951b50a79b709a99b30c66" Nov 03 22:07:29.942550 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.942527 5615 scope.go:122] "RemoveContainer" containerID="f0f6f306addb0899d7820f1419b6b6424c128b0b68fc8f8a7e4b9784a6752d82" Nov 03 22:07:29.953241 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.953220 5615 scope.go:122] "RemoveContainer" containerID="ce6ea373bc6574b8e309d0e99d458bee391a930bd8951b50a79b709a99b30c66" Nov 03 22:07:29.953586 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:07:29.953562 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce6ea373bc6574b8e309d0e99d458bee391a930bd8951b50a79b709a99b30c66\": container with ID starting with ce6ea373bc6574b8e309d0e99d458bee391a930bd8951b50a79b709a99b30c66 not found: ID does not exist" containerID="ce6ea373bc6574b8e309d0e99d458bee391a930bd8951b50a79b709a99b30c66" Nov 03 22:07:29.953702 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.953597 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce6ea373bc6574b8e309d0e99d458bee391a930bd8951b50a79b709a99b30c66"} err="failed to get container status \"ce6ea373bc6574b8e309d0e99d458bee391a930bd8951b50a79b709a99b30c66\": rpc error: code = NotFound desc = could not find container \"ce6ea373bc6574b8e309d0e99d458bee391a930bd8951b50a79b709a99b30c66\": container with ID starting with ce6ea373bc6574b8e309d0e99d458bee391a930bd8951b50a79b709a99b30c66 not found: ID does not exist" Nov 03 22:07:29.953702 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.953621 5615 scope.go:122] "RemoveContainer" containerID="f0f6f306addb0899d7820f1419b6b6424c128b0b68fc8f8a7e4b9784a6752d82" Nov 03 22:07:29.954041 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:07:29.954013 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0f6f306addb0899d7820f1419b6b6424c128b0b68fc8f8a7e4b9784a6752d82\": container with ID starting with f0f6f306addb0899d7820f1419b6b6424c128b0b68fc8f8a7e4b9784a6752d82 not found: ID does not exist" containerID="f0f6f306addb0899d7820f1419b6b6424c128b0b68fc8f8a7e4b9784a6752d82" Nov 03 22:07:29.954191 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:29.954046 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0f6f306addb0899d7820f1419b6b6424c128b0b68fc8f8a7e4b9784a6752d82"} err="failed to get container status \"f0f6f306addb0899d7820f1419b6b6424c128b0b68fc8f8a7e4b9784a6752d82\": rpc error: code = NotFound desc = could not find container \"f0f6f306addb0899d7820f1419b6b6424c128b0b68fc8f8a7e4b9784a6752d82\": container with ID starting with f0f6f306addb0899d7820f1419b6b6424c128b0b68fc8f8a7e4b9784a6752d82 not found: ID does not exist" Nov 03 22:07:30.069298 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:30.069254 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt"] Nov 03 22:07:30.072942 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:30.072910 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-714979-openshift-default-7854774dd9-b4xnt"] Nov 03 22:07:31.651438 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:31.651405 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad38eb17-97aa-4877-9efb-7e324f3ff5bd" path="/var/lib/kubelet/pods/ad38eb17-97aa-4877-9efb-7e324f3ff5bd/volumes" Nov 03 22:07:31.879951 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:31.879905 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s" podUID="cb894762-26e0-4dd6-9e55-0a0aed4455c9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:07:34.552789 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:34.552747 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5"] Nov 03 22:07:34.553344 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:34.552954 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" podUID="88203e05-6890-4c96-9d15-07c6685ad6d3" containerName="istio-proxy" containerID="cri-o://6b8a9aa0533594d108308e2e5a31ffe7afad7e8c33468d2e1e996ff0bb634ac1" gracePeriod=30 Nov 03 22:07:37.202736 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:37.202663 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s" Nov 03 22:07:37.311749 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:37.311711 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb894762-26e0-4dd6-9e55-0a0aed4455c9-kube-api-access-p8v58\" (UniqueName: \"kubernetes.io/projected/cb894762-26e0-4dd6-9e55-0a0aed4455c9-kube-api-access-p8v58\") pod \"cb894762-26e0-4dd6-9e55-0a0aed4455c9\" (UID: \"cb894762-26e0-4dd6-9e55-0a0aed4455c9\") " Nov 03 22:07:37.311949 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:37.311890 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb894762-26e0-4dd6-9e55-0a0aed4455c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb894762-26e0-4dd6-9e55-0a0aed4455c9-plugins-config-volume\") pod \"cb894762-26e0-4dd6-9e55-0a0aed4455c9\" (UID: \"cb894762-26e0-4dd6-9e55-0a0aed4455c9\") " Nov 03 22:07:37.312316 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:37.312282 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb894762-26e0-4dd6-9e55-0a0aed4455c9-plugins-config-volume" pod "cb894762-26e0-4dd6-9e55-0a0aed4455c9" (UID: "cb894762-26e0-4dd6-9e55-0a0aed4455c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:07:37.313838 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:37.313809 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb894762-26e0-4dd6-9e55-0a0aed4455c9-kube-api-access-p8v58" pod "cb894762-26e0-4dd6-9e55-0a0aed4455c9" (UID: "cb894762-26e0-4dd6-9e55-0a0aed4455c9"). InnerVolumeSpecName "kube-api-access-p8v58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:07:37.413584 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:37.413537 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8v58\" (UniqueName: \"kubernetes.io/projected/cb894762-26e0-4dd6-9e55-0a0aed4455c9-kube-api-access-p8v58\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:37.413584 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:37.413576 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb894762-26e0-4dd6-9e55-0a0aed4455c9-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:37.981700 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:37.981604 5615 generic.go:356] "Generic (PLEG): container finished" podID="cb894762-26e0-4dd6-9e55-0a0aed4455c9" containerID="426167e2eb127abef614e6002ae6b0e53dd17cfd61b7e005ca240e1071d557e0" exitCode=0 Nov 03 22:07:37.981700 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:37.981654 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s" Nov 03 22:07:37.981917 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:37.981695 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s" event={"ID":"cb894762-26e0-4dd6-9e55-0a0aed4455c9","Type":"ContainerDied","Data":"426167e2eb127abef614e6002ae6b0e53dd17cfd61b7e005ca240e1071d557e0"} Nov 03 22:07:37.981917 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:37.981754 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s" event={"ID":"cb894762-26e0-4dd6-9e55-0a0aed4455c9","Type":"ContainerDied","Data":"a40a6fd3ddb1e559f816f90dfbbddbd6aec5108fdba741fb596d9d782a991a81"} Nov 03 22:07:37.981917 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:37.981774 5615 scope.go:122] "RemoveContainer" containerID="426167e2eb127abef614e6002ae6b0e53dd17cfd61b7e005ca240e1071d557e0" Nov 03 22:07:37.991201 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:37.991179 5615 scope.go:122] "RemoveContainer" containerID="426167e2eb127abef614e6002ae6b0e53dd17cfd61b7e005ca240e1071d557e0" Nov 03 22:07:37.991479 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:07:37.991458 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"426167e2eb127abef614e6002ae6b0e53dd17cfd61b7e005ca240e1071d557e0\": container with ID starting with 426167e2eb127abef614e6002ae6b0e53dd17cfd61b7e005ca240e1071d557e0 not found: ID does not exist" containerID="426167e2eb127abef614e6002ae6b0e53dd17cfd61b7e005ca240e1071d557e0" Nov 03 22:07:37.991558 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:37.991493 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"426167e2eb127abef614e6002ae6b0e53dd17cfd61b7e005ca240e1071d557e0"} err="failed to get container status \"426167e2eb127abef614e6002ae6b0e53dd17cfd61b7e005ca240e1071d557e0\": rpc error: code = NotFound desc = could not find container \"426167e2eb127abef614e6002ae6b0e53dd17cfd61b7e005ca240e1071d557e0\": container with ID starting with 426167e2eb127abef614e6002ae6b0e53dd17cfd61b7e005ca240e1071d557e0 not found: ID does not exist" Nov 03 22:07:38.148994 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:38.148956 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s"] Nov 03 22:07:38.152925 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:38.152896 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9818d8-65dc594755-tfc5s"] Nov 03 22:07:38.261107 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:38.261061 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a41ad0-8665dcdf7d-948h2"] Nov 03 22:07:38.373542 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:38.373504 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a41ad0-5458b6d47-62h44"] Nov 03 22:07:39.650854 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:39.650823 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb894762-26e0-4dd6-9e55-0a0aed4455c9" path="/var/lib/kubelet/pods/cb894762-26e0-4dd6-9e55-0a0aed4455c9/volumes" Nov 03 22:07:39.853148 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:39.853119 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 22:07:39.943995 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:39.943964 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88203e05-6890-4c96-9d15-07c6685ad6d3-kube-api-access-wvzsk\" (UniqueName: \"kubernetes.io/projected/88203e05-6890-4c96-9d15-07c6685ad6d3-kube-api-access-wvzsk\") pod \"88203e05-6890-4c96-9d15-07c6685ad6d3\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " Nov 03 22:07:39.944195 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:39.944013 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-credential-socket\") pod \"88203e05-6890-4c96-9d15-07c6685ad6d3\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " Nov 03 22:07:39.944195 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:39.944102 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88203e05-6890-4c96-9d15-07c6685ad6d3-istio-token\" (UniqueName: \"kubernetes.io/projected/88203e05-6890-4c96-9d15-07c6685ad6d3-istio-token\") pod \"88203e05-6890-4c96-9d15-07c6685ad6d3\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " Nov 03 22:07:39.944195 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:39.944131 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88203e05-6890-4c96-9d15-07c6685ad6d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88203e05-6890-4c96-9d15-07c6685ad6d3-istio-podinfo\") pod \"88203e05-6890-4c96-9d15-07c6685ad6d3\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " Nov 03 22:07:39.944349 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:39.944228 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-istio-data\") pod \"88203e05-6890-4c96-9d15-07c6685ad6d3\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " Nov 03 22:07:39.944349 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:39.944312 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-workload-socket\") pod \"88203e05-6890-4c96-9d15-07c6685ad6d3\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " Nov 03 22:07:39.944349 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:39.944320 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-credential-socket" pod "88203e05-6890-4c96-9d15-07c6685ad6d3" (UID: "88203e05-6890-4c96-9d15-07c6685ad6d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:39.944514 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:39.944359 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-workload-certs\") pod \"88203e05-6890-4c96-9d15-07c6685ad6d3\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " Nov 03 22:07:39.944514 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:39.944386 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-istio-envoy\") pod \"88203e05-6890-4c96-9d15-07c6685ad6d3\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " Nov 03 22:07:39.944514 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:39.944430 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88203e05-6890-4c96-9d15-07c6685ad6d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88203e05-6890-4c96-9d15-07c6685ad6d3-istiod-ca-cert\") pod \"88203e05-6890-4c96-9d15-07c6685ad6d3\" (UID: \"88203e05-6890-4c96-9d15-07c6685ad6d3\") " Nov 03 22:07:39.944696 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:39.944570 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-istio-data" pod "88203e05-6890-4c96-9d15-07c6685ad6d3" (UID: "88203e05-6890-4c96-9d15-07c6685ad6d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:39.944696 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:39.944591 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-workload-socket" pod "88203e05-6890-4c96-9d15-07c6685ad6d3" (UID: "88203e05-6890-4c96-9d15-07c6685ad6d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:39.944817 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:39.944793 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-workload-certs" pod "88203e05-6890-4c96-9d15-07c6685ad6d3" (UID: "88203e05-6890-4c96-9d15-07c6685ad6d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:39.944925 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:39.944902 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88203e05-6890-4c96-9d15-07c6685ad6d3-istiod-ca-cert" pod "88203e05-6890-4c96-9d15-07c6685ad6d3" (UID: "88203e05-6890-4c96-9d15-07c6685ad6d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:07:39.945003 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:39.944984 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:39.945069 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:39.945010 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:39.945069 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:39.945025 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:39.945069 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:39.945037 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:39.946563 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:39.946538 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88203e05-6890-4c96-9d15-07c6685ad6d3-kube-api-access-wvzsk" pod "88203e05-6890-4c96-9d15-07c6685ad6d3" (UID: "88203e05-6890-4c96-9d15-07c6685ad6d3"). InnerVolumeSpecName "kube-api-access-wvzsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:07:39.946674 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:39.946542 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88203e05-6890-4c96-9d15-07c6685ad6d3-istio-token" pod "88203e05-6890-4c96-9d15-07c6685ad6d3" (UID: "88203e05-6890-4c96-9d15-07c6685ad6d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:07:39.946674 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:39.946616 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88203e05-6890-4c96-9d15-07c6685ad6d3-istio-podinfo" pod "88203e05-6890-4c96-9d15-07c6685ad6d3" (UID: "88203e05-6890-4c96-9d15-07c6685ad6d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:07:39.946674 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:39.946659 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-istio-envoy" pod "88203e05-6890-4c96-9d15-07c6685ad6d3" (UID: "88203e05-6890-4c96-9d15-07c6685ad6d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:39.995382 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:39.995348 5615 generic.go:356] "Generic (PLEG): container finished" podID="88203e05-6890-4c96-9d15-07c6685ad6d3" containerID="6b8a9aa0533594d108308e2e5a31ffe7afad7e8c33468d2e1e996ff0bb634ac1" exitCode=0 Nov 03 22:07:39.995568 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:39.995395 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" Nov 03 22:07:39.995568 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:39.995438 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" event={"ID":"88203e05-6890-4c96-9d15-07c6685ad6d3","Type":"ContainerDied","Data":"6b8a9aa0533594d108308e2e5a31ffe7afad7e8c33468d2e1e996ff0bb634ac1"} Nov 03 22:07:39.995568 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:39.995505 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5" event={"ID":"88203e05-6890-4c96-9d15-07c6685ad6d3","Type":"ContainerDied","Data":"c815e0a612e949d0bed7f36495ae501502d8c7868b6529ea2a4efddf2a3df4cd"} Nov 03 22:07:39.995568 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:39.995534 5615 scope.go:122] "RemoveContainer" containerID="6b8a9aa0533594d108308e2e5a31ffe7afad7e8c33468d2e1e996ff0bb634ac1" Nov 03 22:07:40.005326 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:40.005305 5615 scope.go:122] "RemoveContainer" containerID="1b9eab921b1dfd61588c0517bc335ac9509661386f69c4ff6c9398c4f29cc7ca" Nov 03 22:07:40.014961 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:40.014939 5615 scope.go:122] "RemoveContainer" containerID="6b8a9aa0533594d108308e2e5a31ffe7afad7e8c33468d2e1e996ff0bb634ac1" Nov 03 22:07:40.015268 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:07:40.015248 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b8a9aa0533594d108308e2e5a31ffe7afad7e8c33468d2e1e996ff0bb634ac1\": container with ID starting with 6b8a9aa0533594d108308e2e5a31ffe7afad7e8c33468d2e1e996ff0bb634ac1 not found: ID does not exist" containerID="6b8a9aa0533594d108308e2e5a31ffe7afad7e8c33468d2e1e996ff0bb634ac1" Nov 03 22:07:40.015354 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:40.015283 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b8a9aa0533594d108308e2e5a31ffe7afad7e8c33468d2e1e996ff0bb634ac1"} err="failed to get container status \"6b8a9aa0533594d108308e2e5a31ffe7afad7e8c33468d2e1e996ff0bb634ac1\": rpc error: code = NotFound desc = could not find container \"6b8a9aa0533594d108308e2e5a31ffe7afad7e8c33468d2e1e996ff0bb634ac1\": container with ID starting with 6b8a9aa0533594d108308e2e5a31ffe7afad7e8c33468d2e1e996ff0bb634ac1 not found: ID does not exist" Nov 03 22:07:40.015354 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:40.015309 5615 scope.go:122] "RemoveContainer" containerID="1b9eab921b1dfd61588c0517bc335ac9509661386f69c4ff6c9398c4f29cc7ca" Nov 03 22:07:40.015570 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:07:40.015552 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b9eab921b1dfd61588c0517bc335ac9509661386f69c4ff6c9398c4f29cc7ca\": container with ID starting with 1b9eab921b1dfd61588c0517bc335ac9509661386f69c4ff6c9398c4f29cc7ca not found: ID does not exist" containerID="1b9eab921b1dfd61588c0517bc335ac9509661386f69c4ff6c9398c4f29cc7ca" Nov 03 22:07:40.015621 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:40.015582 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b9eab921b1dfd61588c0517bc335ac9509661386f69c4ff6c9398c4f29cc7ca"} err="failed to get container status \"1b9eab921b1dfd61588c0517bc335ac9509661386f69c4ff6c9398c4f29cc7ca\": rpc error: code = NotFound desc = could not find container \"1b9eab921b1dfd61588c0517bc335ac9509661386f69c4ff6c9398c4f29cc7ca\": container with ID starting with 1b9eab921b1dfd61588c0517bc335ac9509661386f69c4ff6c9398c4f29cc7ca not found: ID does not exist" Nov 03 22:07:40.049145 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:40.048974 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88203e05-6890-4c96-9d15-07c6685ad6d3-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:40.049145 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:40.049012 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88203e05-6890-4c96-9d15-07c6685ad6d3-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:40.049145 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:40.049068 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvzsk\" (UniqueName: \"kubernetes.io/projected/88203e05-6890-4c96-9d15-07c6685ad6d3-kube-api-access-wvzsk\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:40.049145 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:40.049101 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88203e05-6890-4c96-9d15-07c6685ad6d3-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:40.049145 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:40.049114 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88203e05-6890-4c96-9d15-07c6685ad6d3-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:40.148268 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:40.148180 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5"] Nov 03 22:07:40.154210 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:40.154180 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9818d8-openshift-default-f9576f755-bq8c5"] Nov 03 22:07:40.619206 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:07:40.619166 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:07:41.650739 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:41.650704 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88203e05-6890-4c96-9d15-07c6685ad6d3" path="/var/lib/kubelet/pods/88203e05-6890-4c96-9d15-07c6685ad6d3/volumes" Nov 03 22:07:43.262205 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:43.262159 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-a41ad0-8665dcdf7d-948h2" podUID="5611d11d-6bc4-4725-a5e7-d2cf9203f5ce" containerName="payload-processing" containerID="cri-o://81508a73df357b42ccd54bcdfa173bb0298ff9da6234fe73ea1d7cbafe90625f" gracePeriod=25 Nov 03 22:07:43.374853 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:43.374812 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a41ad0-5458b6d47-62h44" podUID="38461555-612e-478c-a9f8-98807b526d0f" containerName="payload-pre-processing" containerID="cri-o://4c5111ca72ef0cf130b501a8f3884341c22777339e3b4b858b7c12c21f93c220" gracePeriod=25 Nov 03 22:07:43.562765 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:43.562723 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a41ad0-8665dcdf7d-948h2" Nov 03 22:07:43.712101 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:43.709563 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a41ad0-5458b6d47-62h44" Nov 03 22:07:43.712101 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:43.711483 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5611d11d-6bc4-4725-a5e7-d2cf9203f5ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5611d11d-6bc4-4725-a5e7-d2cf9203f5ce-plugins-config-volume\") pod \"5611d11d-6bc4-4725-a5e7-d2cf9203f5ce\" (UID: \"5611d11d-6bc4-4725-a5e7-d2cf9203f5ce\") " Nov 03 22:07:43.712101 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:43.711678 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5611d11d-6bc4-4725-a5e7-d2cf9203f5ce-kube-api-access-v55ss\" (UniqueName: \"kubernetes.io/projected/5611d11d-6bc4-4725-a5e7-d2cf9203f5ce-kube-api-access-v55ss\") pod \"5611d11d-6bc4-4725-a5e7-d2cf9203f5ce\" (UID: \"5611d11d-6bc4-4725-a5e7-d2cf9203f5ce\") " Nov 03 22:07:43.714845 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:43.714750 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5611d11d-6bc4-4725-a5e7-d2cf9203f5ce-plugins-config-volume" pod "5611d11d-6bc4-4725-a5e7-d2cf9203f5ce" (UID: "5611d11d-6bc4-4725-a5e7-d2cf9203f5ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:07:43.720416 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:43.720381 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5611d11d-6bc4-4725-a5e7-d2cf9203f5ce-kube-api-access-v55ss" pod "5611d11d-6bc4-4725-a5e7-d2cf9203f5ce" (UID: "5611d11d-6bc4-4725-a5e7-d2cf9203f5ce"). InnerVolumeSpecName "kube-api-access-v55ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:07:43.817999 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:43.817893 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38461555-612e-478c-a9f8-98807b526d0f-kube-api-access-nnn85\" (UniqueName: \"kubernetes.io/projected/38461555-612e-478c-a9f8-98807b526d0f-kube-api-access-nnn85\") pod \"38461555-612e-478c-a9f8-98807b526d0f\" (UID: \"38461555-612e-478c-a9f8-98807b526d0f\") " Nov 03 22:07:43.817999 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:43.817969 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38461555-612e-478c-a9f8-98807b526d0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38461555-612e-478c-a9f8-98807b526d0f-plugins-config-volume\") pod \"38461555-612e-478c-a9f8-98807b526d0f\" (UID: \"38461555-612e-478c-a9f8-98807b526d0f\") " Nov 03 22:07:43.818417 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:43.818397 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v55ss\" (UniqueName: \"kubernetes.io/projected/5611d11d-6bc4-4725-a5e7-d2cf9203f5ce-kube-api-access-v55ss\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:43.818417 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:43.818415 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5611d11d-6bc4-4725-a5e7-d2cf9203f5ce-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:43.818788 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:43.818759 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38461555-612e-478c-a9f8-98807b526d0f-plugins-config-volume" pod "38461555-612e-478c-a9f8-98807b526d0f" (UID: "38461555-612e-478c-a9f8-98807b526d0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:07:43.841643 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:43.836317 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38461555-612e-478c-a9f8-98807b526d0f-kube-api-access-nnn85" pod "38461555-612e-478c-a9f8-98807b526d0f" (UID: "38461555-612e-478c-a9f8-98807b526d0f"). InnerVolumeSpecName "kube-api-access-nnn85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:07:43.919809 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:43.919771 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38461555-612e-478c-a9f8-98807b526d0f-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:43.919809 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:43.919804 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnn85\" (UniqueName: \"kubernetes.io/projected/38461555-612e-478c-a9f8-98807b526d0f-kube-api-access-nnn85\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:44.019335 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:44.019297 5615 generic.go:356] "Generic (PLEG): container finished" podID="5611d11d-6bc4-4725-a5e7-d2cf9203f5ce" containerID="81508a73df357b42ccd54bcdfa173bb0298ff9da6234fe73ea1d7cbafe90625f" exitCode=0 Nov 03 22:07:44.019531 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:44.019340 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a41ad0-8665dcdf7d-948h2" Nov 03 22:07:44.019531 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:44.019385 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a41ad0-8665dcdf7d-948h2" event={"ID":"5611d11d-6bc4-4725-a5e7-d2cf9203f5ce","Type":"ContainerDied","Data":"81508a73df357b42ccd54bcdfa173bb0298ff9da6234fe73ea1d7cbafe90625f"} Nov 03 22:07:44.019531 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:44.019441 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a41ad0-8665dcdf7d-948h2" event={"ID":"5611d11d-6bc4-4725-a5e7-d2cf9203f5ce","Type":"ContainerDied","Data":"665b39d1c295abd581b016aedffb80288665515f41de56dd518a2eeb21f25791"} Nov 03 22:07:44.019531 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:44.019464 5615 scope.go:122] "RemoveContainer" containerID="81508a73df357b42ccd54bcdfa173bb0298ff9da6234fe73ea1d7cbafe90625f" Nov 03 22:07:44.020901 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:44.020879 5615 generic.go:356] "Generic (PLEG): container finished" podID="38461555-612e-478c-a9f8-98807b526d0f" containerID="4c5111ca72ef0cf130b501a8f3884341c22777339e3b4b858b7c12c21f93c220" exitCode=0 Nov 03 22:07:44.021000 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:44.020926 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a41ad0-5458b6d47-62h44" Nov 03 22:07:44.021000 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:44.020965 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a41ad0-5458b6d47-62h44" event={"ID":"38461555-612e-478c-a9f8-98807b526d0f","Type":"ContainerDied","Data":"4c5111ca72ef0cf130b501a8f3884341c22777339e3b4b858b7c12c21f93c220"} Nov 03 22:07:44.021163 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:44.021016 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a41ad0-5458b6d47-62h44" event={"ID":"38461555-612e-478c-a9f8-98807b526d0f","Type":"ContainerDied","Data":"4eeeff456f0eb0c6316a773a93a50c018426ad32350b51cf072a41dc1294f6b7"} Nov 03 22:07:44.030429 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:44.030124 5615 scope.go:122] "RemoveContainer" containerID="81508a73df357b42ccd54bcdfa173bb0298ff9da6234fe73ea1d7cbafe90625f" Nov 03 22:07:44.030541 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:07:44.030492 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81508a73df357b42ccd54bcdfa173bb0298ff9da6234fe73ea1d7cbafe90625f\": container with ID starting with 81508a73df357b42ccd54bcdfa173bb0298ff9da6234fe73ea1d7cbafe90625f not found: ID does not exist" containerID="81508a73df357b42ccd54bcdfa173bb0298ff9da6234fe73ea1d7cbafe90625f" Nov 03 22:07:44.030599 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:44.030525 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81508a73df357b42ccd54bcdfa173bb0298ff9da6234fe73ea1d7cbafe90625f"} err="failed to get container status \"81508a73df357b42ccd54bcdfa173bb0298ff9da6234fe73ea1d7cbafe90625f\": rpc error: code = NotFound desc = could not find container \"81508a73df357b42ccd54bcdfa173bb0298ff9da6234fe73ea1d7cbafe90625f\": container with ID starting with 81508a73df357b42ccd54bcdfa173bb0298ff9da6234fe73ea1d7cbafe90625f not found: ID does not exist" Nov 03 22:07:44.030599 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:44.030551 5615 scope.go:122] "RemoveContainer" containerID="4c5111ca72ef0cf130b501a8f3884341c22777339e3b4b858b7c12c21f93c220" Nov 03 22:07:44.039270 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:44.039247 5615 scope.go:122] "RemoveContainer" containerID="4c5111ca72ef0cf130b501a8f3884341c22777339e3b4b858b7c12c21f93c220" Nov 03 22:07:44.039531 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:07:44.039504 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c5111ca72ef0cf130b501a8f3884341c22777339e3b4b858b7c12c21f93c220\": container with ID starting with 4c5111ca72ef0cf130b501a8f3884341c22777339e3b4b858b7c12c21f93c220 not found: ID does not exist" containerID="4c5111ca72ef0cf130b501a8f3884341c22777339e3b4b858b7c12c21f93c220" Nov 03 22:07:44.039576 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:44.039533 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c5111ca72ef0cf130b501a8f3884341c22777339e3b4b858b7c12c21f93c220"} err="failed to get container status \"4c5111ca72ef0cf130b501a8f3884341c22777339e3b4b858b7c12c21f93c220\": rpc error: code = NotFound desc = could not find container \"4c5111ca72ef0cf130b501a8f3884341c22777339e3b4b858b7c12c21f93c220\": container with ID starting with 4c5111ca72ef0cf130b501a8f3884341c22777339e3b4b858b7c12c21f93c220 not found: ID does not exist" Nov 03 22:07:44.194823 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:44.194736 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a41ad0-8665dcdf7d-948h2"] Nov 03 22:07:44.199492 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:44.199464 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a41ad0-8665dcdf7d-948h2"] Nov 03 22:07:44.311668 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:44.311629 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a41ad0-5458b6d47-62h44"] Nov 03 22:07:44.319085 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:44.319055 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a41ad0-5458b6d47-62h44"] Nov 03 22:07:44.687176 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:44.687119 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" podUID="39c75c51-eff9-4cb8-88b9-618dbb2ba188" containerName="main" containerID="cri-o://9726c702cffdc192c8b51ce7a0125d5609839491dae2879630182a0a4d175614" gracePeriod=45 Nov 03 22:07:44.701220 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:44.701181 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" podUID="a89b0627-f993-4003-ac8b-fc449b44d9d9" containerName="main" containerID="cri-o://cd9400e70c37c0078b7bc57f940c1448918fd6bd6f7a48c9902fdd2145238309" gracePeriod=45 Nov 03 22:07:44.992727 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:44.992701 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" Nov 03 22:07:44.995955 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:44.995925 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" Nov 03 22:07:45.028565 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.028532 5615 generic.go:356] "Generic (PLEG): container finished" podID="39c75c51-eff9-4cb8-88b9-618dbb2ba188" containerID="9726c702cffdc192c8b51ce7a0125d5609839491dae2879630182a0a4d175614" exitCode=0 Nov 03 22:07:45.028823 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.028594 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" Nov 03 22:07:45.028823 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.028616 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" event={"ID":"39c75c51-eff9-4cb8-88b9-618dbb2ba188","Type":"ContainerDied","Data":"9726c702cffdc192c8b51ce7a0125d5609839491dae2879630182a0a4d175614"} Nov 03 22:07:45.028823 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.028659 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6" event={"ID":"39c75c51-eff9-4cb8-88b9-618dbb2ba188","Type":"ContainerDied","Data":"63f7aa919b27c56c9f2949e93b478877903ef25da169174508fed4ea9fa235a0"} Nov 03 22:07:45.028823 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.028679 5615 scope.go:122] "RemoveContainer" containerID="9726c702cffdc192c8b51ce7a0125d5609839491dae2879630182a0a4d175614" Nov 03 22:07:45.031725 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.031690 5615 generic.go:356] "Generic (PLEG): container finished" podID="a89b0627-f993-4003-ac8b-fc449b44d9d9" containerID="cd9400e70c37c0078b7bc57f940c1448918fd6bd6f7a48c9902fdd2145238309" exitCode=0 Nov 03 22:07:45.031906 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.031851 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" event={"ID":"a89b0627-f993-4003-ac8b-fc449b44d9d9","Type":"ContainerDied","Data":"cd9400e70c37c0078b7bc57f940c1448918fd6bd6f7a48c9902fdd2145238309"} Nov 03 22:07:45.031906 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.031876 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" Nov 03 22:07:45.032041 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.031906 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" event={"ID":"a89b0627-f993-4003-ac8b-fc449b44d9d9","Type":"ContainerDied","Data":"1c6e6ba78c400559800abda8c9c553a5156a04e3b17e2f0b0c41bd76712c8ea5"} Nov 03 22:07:45.049312 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.049274 5615 scope.go:122] "RemoveContainer" containerID="9726c702cffdc192c8b51ce7a0125d5609839491dae2879630182a0a4d175614" Nov 03 22:07:45.049760 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:07:45.049727 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9726c702cffdc192c8b51ce7a0125d5609839491dae2879630182a0a4d175614\": container with ID starting with 9726c702cffdc192c8b51ce7a0125d5609839491dae2879630182a0a4d175614 not found: ID does not exist" containerID="9726c702cffdc192c8b51ce7a0125d5609839491dae2879630182a0a4d175614" Nov 03 22:07:45.049859 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.049799 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9726c702cffdc192c8b51ce7a0125d5609839491dae2879630182a0a4d175614"} err="failed to get container status \"9726c702cffdc192c8b51ce7a0125d5609839491dae2879630182a0a4d175614\": rpc error: code = NotFound desc = could not find container \"9726c702cffdc192c8b51ce7a0125d5609839491dae2879630182a0a4d175614\": container with ID starting with 9726c702cffdc192c8b51ce7a0125d5609839491dae2879630182a0a4d175614 not found: ID does not exist" Nov 03 22:07:45.049859 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.049846 5615 scope.go:122] "RemoveContainer" containerID="cd9400e70c37c0078b7bc57f940c1448918fd6bd6f7a48c9902fdd2145238309" Nov 03 22:07:45.058487 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.058466 5615 scope.go:122] "RemoveContainer" containerID="cd9400e70c37c0078b7bc57f940c1448918fd6bd6f7a48c9902fdd2145238309" Nov 03 22:07:45.058785 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:07:45.058762 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd9400e70c37c0078b7bc57f940c1448918fd6bd6f7a48c9902fdd2145238309\": container with ID starting with cd9400e70c37c0078b7bc57f940c1448918fd6bd6f7a48c9902fdd2145238309 not found: ID does not exist" containerID="cd9400e70c37c0078b7bc57f940c1448918fd6bd6f7a48c9902fdd2145238309" Nov 03 22:07:45.058830 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.058796 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd9400e70c37c0078b7bc57f940c1448918fd6bd6f7a48c9902fdd2145238309"} err="failed to get container status \"cd9400e70c37c0078b7bc57f940c1448918fd6bd6f7a48c9902fdd2145238309\": rpc error: code = NotFound desc = could not find container \"cd9400e70c37c0078b7bc57f940c1448918fd6bd6f7a48c9902fdd2145238309\": container with ID starting with cd9400e70c37c0078b7bc57f940c1448918fd6bd6f7a48c9902fdd2145238309 not found: ID does not exist" Nov 03 22:07:45.132620 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.132590 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a89b0627-f993-4003-ac8b-fc449b44d9d9-home\" (UniqueName: \"kubernetes.io/empty-dir/a89b0627-f993-4003-ac8b-fc449b44d9d9-home\") pod \"a89b0627-f993-4003-ac8b-fc449b44d9d9\" (UID: \"a89b0627-f993-4003-ac8b-fc449b44d9d9\") " Nov 03 22:07:45.132776 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.132643 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a89b0627-f993-4003-ac8b-fc449b44d9d9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a89b0627-f993-4003-ac8b-fc449b44d9d9-tmp-dir\") pod \"a89b0627-f993-4003-ac8b-fc449b44d9d9\" (UID: \"a89b0627-f993-4003-ac8b-fc449b44d9d9\") " Nov 03 22:07:45.132776 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.132704 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a89b0627-f993-4003-ac8b-fc449b44d9d9-kube-api-access-cz2tz\" (UniqueName: \"kubernetes.io/projected/a89b0627-f993-4003-ac8b-fc449b44d9d9-kube-api-access-cz2tz\") pod \"a89b0627-f993-4003-ac8b-fc449b44d9d9\" (UID: \"a89b0627-f993-4003-ac8b-fc449b44d9d9\") " Nov 03 22:07:45.132776 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.132757 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a89b0627-f993-4003-ac8b-fc449b44d9d9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a89b0627-f993-4003-ac8b-fc449b44d9d9-model-cache\") pod \"a89b0627-f993-4003-ac8b-fc449b44d9d9\" (UID: \"a89b0627-f993-4003-ac8b-fc449b44d9d9\") " Nov 03 22:07:45.132919 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.132812 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39c75c51-eff9-4cb8-88b9-618dbb2ba188-kube-api-access-7ccrs\" (UniqueName: \"kubernetes.io/projected/39c75c51-eff9-4cb8-88b9-618dbb2ba188-kube-api-access-7ccrs\") pod \"39c75c51-eff9-4cb8-88b9-618dbb2ba188\" (UID: \"39c75c51-eff9-4cb8-88b9-618dbb2ba188\") " Nov 03 22:07:45.132919 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.132845 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39c75c51-eff9-4cb8-88b9-618dbb2ba188-dshm\" (UniqueName: \"kubernetes.io/empty-dir/39c75c51-eff9-4cb8-88b9-618dbb2ba188-dshm\") pod \"39c75c51-eff9-4cb8-88b9-618dbb2ba188\" (UID: \"39c75c51-eff9-4cb8-88b9-618dbb2ba188\") " Nov 03 22:07:45.132919 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.132883 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a89b0627-f993-4003-ac8b-fc449b44d9d9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a89b0627-f993-4003-ac8b-fc449b44d9d9-dshm\") pod \"a89b0627-f993-4003-ac8b-fc449b44d9d9\" (UID: \"a89b0627-f993-4003-ac8b-fc449b44d9d9\") " Nov 03 22:07:45.133090 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.132904 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89b0627-f993-4003-ac8b-fc449b44d9d9-home" pod "a89b0627-f993-4003-ac8b-fc449b44d9d9" (UID: "a89b0627-f993-4003-ac8b-fc449b44d9d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:45.133090 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.132979 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39c75c51-eff9-4cb8-88b9-618dbb2ba188-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39c75c51-eff9-4cb8-88b9-618dbb2ba188-model-cache\") pod \"39c75c51-eff9-4cb8-88b9-618dbb2ba188\" (UID: \"39c75c51-eff9-4cb8-88b9-618dbb2ba188\") " Nov 03 22:07:45.133090 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.132975 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89b0627-f993-4003-ac8b-fc449b44d9d9-tmp-dir" pod "a89b0627-f993-4003-ac8b-fc449b44d9d9" (UID: "a89b0627-f993-4003-ac8b-fc449b44d9d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:45.133090 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.133029 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a89b0627-f993-4003-ac8b-fc449b44d9d9-tls-certs\" (UniqueName: \"kubernetes.io/secret/a89b0627-f993-4003-ac8b-fc449b44d9d9-tls-certs\") pod \"a89b0627-f993-4003-ac8b-fc449b44d9d9\" (UID: \"a89b0627-f993-4003-ac8b-fc449b44d9d9\") " Nov 03 22:07:45.133090 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.133048 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39c75c51-eff9-4cb8-88b9-618dbb2ba188-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39c75c51-eff9-4cb8-88b9-618dbb2ba188-tmp-dir\") pod \"39c75c51-eff9-4cb8-88b9-618dbb2ba188\" (UID: \"39c75c51-eff9-4cb8-88b9-618dbb2ba188\") " Nov 03 22:07:45.133386 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.133099 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39c75c51-eff9-4cb8-88b9-618dbb2ba188-home\" (UniqueName: \"kubernetes.io/empty-dir/39c75c51-eff9-4cb8-88b9-618dbb2ba188-home\") pod \"39c75c51-eff9-4cb8-88b9-618dbb2ba188\" (UID: \"39c75c51-eff9-4cb8-88b9-618dbb2ba188\") " Nov 03 22:07:45.133386 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.133128 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89b0627-f993-4003-ac8b-fc449b44d9d9-model-cache" pod "a89b0627-f993-4003-ac8b-fc449b44d9d9" (UID: "a89b0627-f993-4003-ac8b-fc449b44d9d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:45.133386 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.133145 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39c75c51-eff9-4cb8-88b9-618dbb2ba188-tls-certs\" (UniqueName: \"kubernetes.io/secret/39c75c51-eff9-4cb8-88b9-618dbb2ba188-tls-certs\") pod \"39c75c51-eff9-4cb8-88b9-618dbb2ba188\" (UID: \"39c75c51-eff9-4cb8-88b9-618dbb2ba188\") " Nov 03 22:07:45.134224 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.133754 5615 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a89b0627-f993-4003-ac8b-fc449b44d9d9-home\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:45.134224 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.133826 5615 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a89b0627-f993-4003-ac8b-fc449b44d9d9-tmp-dir\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:45.134224 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.133846 5615 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a89b0627-f993-4003-ac8b-fc449b44d9d9-model-cache\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:45.134224 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.133831 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c75c51-eff9-4cb8-88b9-618dbb2ba188-model-cache" pod "39c75c51-eff9-4cb8-88b9-618dbb2ba188" (UID: "39c75c51-eff9-4cb8-88b9-618dbb2ba188"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:45.134224 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.133855 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c75c51-eff9-4cb8-88b9-618dbb2ba188-home" pod "39c75c51-eff9-4cb8-88b9-618dbb2ba188" (UID: "39c75c51-eff9-4cb8-88b9-618dbb2ba188"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:45.134224 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.134136 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c75c51-eff9-4cb8-88b9-618dbb2ba188-tmp-dir" pod "39c75c51-eff9-4cb8-88b9-618dbb2ba188" (UID: "39c75c51-eff9-4cb8-88b9-618dbb2ba188"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:45.136011 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.135972 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c75c51-eff9-4cb8-88b9-618dbb2ba188-kube-api-access-7ccrs" pod "39c75c51-eff9-4cb8-88b9-618dbb2ba188" (UID: "39c75c51-eff9-4cb8-88b9-618dbb2ba188"). InnerVolumeSpecName "kube-api-access-7ccrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:07:45.136207 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.136101 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89b0627-f993-4003-ac8b-fc449b44d9d9-dshm" pod "a89b0627-f993-4003-ac8b-fc449b44d9d9" (UID: "a89b0627-f993-4003-ac8b-fc449b44d9d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:45.136327 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.136306 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a89b0627-f993-4003-ac8b-fc449b44d9d9-tls-certs" pod "a89b0627-f993-4003-ac8b-fc449b44d9d9" (UID: "a89b0627-f993-4003-ac8b-fc449b44d9d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:07:45.136553 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.136526 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a89b0627-f993-4003-ac8b-fc449b44d9d9-kube-api-access-cz2tz" pod "a89b0627-f993-4003-ac8b-fc449b44d9d9" (UID: "a89b0627-f993-4003-ac8b-fc449b44d9d9"). InnerVolumeSpecName "kube-api-access-cz2tz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:07:45.136772 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.136753 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39c75c51-eff9-4cb8-88b9-618dbb2ba188-tls-certs" pod "39c75c51-eff9-4cb8-88b9-618dbb2ba188" (UID: "39c75c51-eff9-4cb8-88b9-618dbb2ba188"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:07:45.137158 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.137138 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c75c51-eff9-4cb8-88b9-618dbb2ba188-dshm" pod "39c75c51-eff9-4cb8-88b9-618dbb2ba188" (UID: "39c75c51-eff9-4cb8-88b9-618dbb2ba188"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:45.235196 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.235167 5615 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a89b0627-f993-4003-ac8b-fc449b44d9d9-tls-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:45.235196 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.235195 5615 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39c75c51-eff9-4cb8-88b9-618dbb2ba188-tmp-dir\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:45.235389 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.235205 5615 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39c75c51-eff9-4cb8-88b9-618dbb2ba188-home\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:45.235389 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.235215 5615 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39c75c51-eff9-4cb8-88b9-618dbb2ba188-tls-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:45.235389 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.235224 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz2tz\" (UniqueName: \"kubernetes.io/projected/a89b0627-f993-4003-ac8b-fc449b44d9d9-kube-api-access-cz2tz\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:45.235389 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.235234 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ccrs\" (UniqueName: \"kubernetes.io/projected/39c75c51-eff9-4cb8-88b9-618dbb2ba188-kube-api-access-7ccrs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:45.235389 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.235243 5615 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39c75c51-eff9-4cb8-88b9-618dbb2ba188-dshm\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:45.235389 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.235250 5615 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a89b0627-f993-4003-ac8b-fc449b44d9d9-dshm\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:45.235389 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.235258 5615 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39c75c51-eff9-4cb8-88b9-618dbb2ba188-model-cache\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:45.473343 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.473306 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6"] Nov 03 22:07:45.501913 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.496005 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a41ad0/distinct-w1-a41ad0-kserve-7d4655b5b7-lp4l6"] Nov 03 22:07:45.607006 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.606971 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz"] Nov 03 22:07:45.614726 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.614687 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz"] Nov 03 22:07:45.651664 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.651631 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38461555-612e-478c-a9f8-98807b526d0f" path="/var/lib/kubelet/pods/38461555-612e-478c-a9f8-98807b526d0f/volumes" Nov 03 22:07:45.652063 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.652049 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39c75c51-eff9-4cb8-88b9-618dbb2ba188" path="/var/lib/kubelet/pods/39c75c51-eff9-4cb8-88b9-618dbb2ba188/volumes" Nov 03 22:07:45.652425 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.652411 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5611d11d-6bc4-4725-a5e7-d2cf9203f5ce" path="/var/lib/kubelet/pods/5611d11d-6bc4-4725-a5e7-d2cf9203f5ce/volumes" Nov 03 22:07:45.653096 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.653066 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a89b0627-f993-4003-ac8b-fc449b44d9d9" path="/var/lib/kubelet/pods/a89b0627-f993-4003-ac8b-fc449b44d9d9/volumes" Nov 03 22:07:45.920105 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:07:45.919968 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a41ad0/simulator-w1-a41ad0-kserve-6d57df497-n2cjz" podUID="a89b0627-f993-4003-ac8b-fc449b44d9d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 03 22:08:07.520238 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:07.520155 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r"] Nov 03 22:08:07.520731 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:07.520423 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r" podUID="614c1da8-3cb5-4362-8774-b6b76ab79ac2" containerName="maas-api" containerID="cri-o://36828cfd137c91db4b814b0520e8cb019965e5d2d6f3784157ded54a609bc2e6" gracePeriod=30 Nov 03 22:08:07.797322 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:07.797287 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r" Nov 03 22:08:07.893891 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:07.889427 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/614c1da8-3cb5-4362-8774-b6b76ab79ac2-kube-api-access-5k7r2\" (UniqueName: \"kubernetes.io/projected/614c1da8-3cb5-4362-8774-b6b76ab79ac2-kube-api-access-5k7r2\") pod \"614c1da8-3cb5-4362-8774-b6b76ab79ac2\" (UID: \"614c1da8-3cb5-4362-8774-b6b76ab79ac2\") " Nov 03 22:08:07.893891 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:07.889523 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/614c1da8-3cb5-4362-8774-b6b76ab79ac2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/614c1da8-3cb5-4362-8774-b6b76ab79ac2-metrics-certs\") pod \"614c1da8-3cb5-4362-8774-b6b76ab79ac2\" (UID: \"614c1da8-3cb5-4362-8774-b6b76ab79ac2\") " Nov 03 22:08:07.893891 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:07.889670 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/614c1da8-3cb5-4362-8774-b6b76ab79ac2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/614c1da8-3cb5-4362-8774-b6b76ab79ac2-maas-api-tls\") pod \"614c1da8-3cb5-4362-8774-b6b76ab79ac2\" (UID: \"614c1da8-3cb5-4362-8774-b6b76ab79ac2\") " Nov 03 22:08:07.893891 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:07.889852 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/614c1da8-3cb5-4362-8774-b6b76ab79ac2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/614c1da8-3cb5-4362-8774-b6b76ab79ac2-openshift-service-ca\") pod \"614c1da8-3cb5-4362-8774-b6b76ab79ac2\" (UID: \"614c1da8-3cb5-4362-8774-b6b76ab79ac2\") " Nov 03 22:08:07.893891 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:07.890989 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/614c1da8-3cb5-4362-8774-b6b76ab79ac2-openshift-service-ca" pod "614c1da8-3cb5-4362-8774-b6b76ab79ac2" (UID: "614c1da8-3cb5-4362-8774-b6b76ab79ac2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:08:07.895527 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:07.895481 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/614c1da8-3cb5-4362-8774-b6b76ab79ac2-metrics-certs" pod "614c1da8-3cb5-4362-8774-b6b76ab79ac2" (UID: "614c1da8-3cb5-4362-8774-b6b76ab79ac2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:08:07.895734 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:07.895556 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/614c1da8-3cb5-4362-8774-b6b76ab79ac2-maas-api-tls" pod "614c1da8-3cb5-4362-8774-b6b76ab79ac2" (UID: "614c1da8-3cb5-4362-8774-b6b76ab79ac2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:08:07.895993 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:07.895952 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/614c1da8-3cb5-4362-8774-b6b76ab79ac2-kube-api-access-5k7r2" pod "614c1da8-3cb5-4362-8774-b6b76ab79ac2" (UID: "614c1da8-3cb5-4362-8774-b6b76ab79ac2"). InnerVolumeSpecName "kube-api-access-5k7r2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:07.991435 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:07.991381 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/614c1da8-3cb5-4362-8774-b6b76ab79ac2-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:07.991435 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:07.991431 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k7r2\" (UniqueName: \"kubernetes.io/projected/614c1da8-3cb5-4362-8774-b6b76ab79ac2-kube-api-access-5k7r2\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:07.991435 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:07.991443 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/614c1da8-3cb5-4362-8774-b6b76ab79ac2-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:07.991435 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:07.991454 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/614c1da8-3cb5-4362-8774-b6b76ab79ac2-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:08.160459 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:08.160377 5615 generic.go:356] "Generic (PLEG): container finished" podID="614c1da8-3cb5-4362-8774-b6b76ab79ac2" containerID="36828cfd137c91db4b814b0520e8cb019965e5d2d6f3784157ded54a609bc2e6" exitCode=0 Nov 03 22:08:08.160459 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:08.160429 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r" Nov 03 22:08:08.160639 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:08.160465 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r" event={"ID":"614c1da8-3cb5-4362-8774-b6b76ab79ac2","Type":"ContainerDied","Data":"36828cfd137c91db4b814b0520e8cb019965e5d2d6f3784157ded54a609bc2e6"} Nov 03 22:08:08.160639 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:08.160522 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r" event={"ID":"614c1da8-3cb5-4362-8774-b6b76ab79ac2","Type":"ContainerDied","Data":"cf319282cd8c6b1231351a10f200bc290b0166ebcb4a453817016408686c49f0"} Nov 03 22:08:08.160639 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:08.160549 5615 scope.go:122] "RemoveContainer" containerID="36828cfd137c91db4b814b0520e8cb019965e5d2d6f3784157ded54a609bc2e6" Nov 03 22:08:08.170946 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:08.170926 5615 scope.go:122] "RemoveContainer" containerID="36828cfd137c91db4b814b0520e8cb019965e5d2d6f3784157ded54a609bc2e6" Nov 03 22:08:08.171258 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:08:08.171236 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36828cfd137c91db4b814b0520e8cb019965e5d2d6f3784157ded54a609bc2e6\": container with ID starting with 36828cfd137c91db4b814b0520e8cb019965e5d2d6f3784157ded54a609bc2e6 not found: ID does not exist" containerID="36828cfd137c91db4b814b0520e8cb019965e5d2d6f3784157ded54a609bc2e6" Nov 03 22:08:08.171313 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:08.171268 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36828cfd137c91db4b814b0520e8cb019965e5d2d6f3784157ded54a609bc2e6"} err="failed to get container status \"36828cfd137c91db4b814b0520e8cb019965e5d2d6f3784157ded54a609bc2e6\": rpc error: code = NotFound desc = could not find container \"36828cfd137c91db4b814b0520e8cb019965e5d2d6f3784157ded54a609bc2e6\": container with ID starting with 36828cfd137c91db4b814b0520e8cb019965e5d2d6f3784157ded54a609bc2e6 not found: ID does not exist" Nov 03 22:08:08.329025 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:08.328982 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r"] Nov 03 22:08:08.333569 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:08.333534 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a6219b-797487b796-jvc2r"] Nov 03 22:08:09.651420 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:09.651384 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="614c1da8-3cb5-4362-8774-b6b76ab79ac2" path="/var/lib/kubelet/pods/614c1da8-3cb5-4362-8774-b6b76ab79ac2/volumes" Nov 03 22:08:10.617998 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:08:10.617957 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:08:13.618371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:13.618334 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg"] Nov 03 22:08:13.618851 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:13.618552 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" podUID="0fd70b58-6f75-415c-b26e-303faf172e6c" containerName="istio-proxy" containerID="cri-o://136f7114eef954168a0ef368c9a05ca5f7c7659623bb98a30aaa963bc13d1777" gracePeriod=30 Nov 03 22:08:17.333339 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:17.333299 5615 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:08:17.333748 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:17.333371 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" podUID="0fd70b58-6f75-415c-b26e-303faf172e6c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:08:18.420481 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:18.420443 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t"] Nov 03 22:08:18.420875 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:18.420711 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" podUID="a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b" containerName="istio-proxy" containerID="cri-o://1e92f3208be893b817dd50dbc19bc41713288b24dbb2c2a80fd7c5a495e63e94" gracePeriod=30 Nov 03 22:08:18.881149 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:18.881117 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 22:08:19.028765 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.028682 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-workload-certs\") pod \"0fd70b58-6f75-415c-b26e-303faf172e6c\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " Nov 03 22:08:19.028765 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.028751 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0fd70b58-6f75-415c-b26e-303faf172e6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fd70b58-6f75-415c-b26e-303faf172e6c-istio-podinfo\") pod \"0fd70b58-6f75-415c-b26e-303faf172e6c\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " Nov 03 22:08:19.029012 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.028789 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fd70b58-6f75-415c-b26e-303faf172e6c-kube-api-access-89lmc\" (UniqueName: \"kubernetes.io/projected/0fd70b58-6f75-415c-b26e-303faf172e6c-kube-api-access-89lmc\") pod \"0fd70b58-6f75-415c-b26e-303faf172e6c\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " Nov 03 22:08:19.029012 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.028835 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-istio-envoy\") pod \"0fd70b58-6f75-415c-b26e-303faf172e6c\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " Nov 03 22:08:19.029012 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.028865 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fd70b58-6f75-415c-b26e-303faf172e6c-istio-token\" (UniqueName: \"kubernetes.io/projected/0fd70b58-6f75-415c-b26e-303faf172e6c-istio-token\") pod \"0fd70b58-6f75-415c-b26e-303faf172e6c\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " Nov 03 22:08:19.029012 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.028957 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-credential-socket\") pod \"0fd70b58-6f75-415c-b26e-303faf172e6c\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " Nov 03 22:08:19.029012 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.028993 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-workload-certs" pod "0fd70b58-6f75-415c-b26e-303faf172e6c" (UID: "0fd70b58-6f75-415c-b26e-303faf172e6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:19.029365 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.028999 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fd70b58-6f75-415c-b26e-303faf172e6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fd70b58-6f75-415c-b26e-303faf172e6c-istiod-ca-cert\") pod \"0fd70b58-6f75-415c-b26e-303faf172e6c\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " Nov 03 22:08:19.029365 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.029051 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-workload-socket\") pod \"0fd70b58-6f75-415c-b26e-303faf172e6c\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " Nov 03 22:08:19.029365 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.029180 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-istio-data\") pod \"0fd70b58-6f75-415c-b26e-303faf172e6c\" (UID: \"0fd70b58-6f75-415c-b26e-303faf172e6c\") " Nov 03 22:08:19.029741 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.029652 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:19.029830 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.029793 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-workload-socket" pod "0fd70b58-6f75-415c-b26e-303faf172e6c" (UID: "0fd70b58-6f75-415c-b26e-303faf172e6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:19.029891 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.029828 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-istio-data" pod "0fd70b58-6f75-415c-b26e-303faf172e6c" (UID: "0fd70b58-6f75-415c-b26e-303faf172e6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:19.030027 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.029987 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-credential-socket" pod "0fd70b58-6f75-415c-b26e-303faf172e6c" (UID: "0fd70b58-6f75-415c-b26e-303faf172e6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:19.030125 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.030087 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fd70b58-6f75-415c-b26e-303faf172e6c-istiod-ca-cert" pod "0fd70b58-6f75-415c-b26e-303faf172e6c" (UID: "0fd70b58-6f75-415c-b26e-303faf172e6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:08:19.031671 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.031636 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0fd70b58-6f75-415c-b26e-303faf172e6c-istio-podinfo" pod "0fd70b58-6f75-415c-b26e-303faf172e6c" (UID: "0fd70b58-6f75-415c-b26e-303faf172e6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:08:19.032006 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.031984 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd70b58-6f75-415c-b26e-303faf172e6c-kube-api-access-89lmc" pod "0fd70b58-6f75-415c-b26e-303faf172e6c" (UID: "0fd70b58-6f75-415c-b26e-303faf172e6c"). InnerVolumeSpecName "kube-api-access-89lmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:19.032168 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.032145 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-istio-envoy" pod "0fd70b58-6f75-415c-b26e-303faf172e6c" (UID: "0fd70b58-6f75-415c-b26e-303faf172e6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:19.032219 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.032176 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd70b58-6f75-415c-b26e-303faf172e6c-istio-token" pod "0fd70b58-6f75-415c-b26e-303faf172e6c" (UID: "0fd70b58-6f75-415c-b26e-303faf172e6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:19.130817 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.130781 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:19.130817 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.130811 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fd70b58-6f75-415c-b26e-303faf172e6c-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:19.130817 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.130821 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:19.131099 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.130830 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:19.131099 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.130840 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fd70b58-6f75-415c-b26e-303faf172e6c-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:19.131099 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.130851 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89lmc\" (UniqueName: \"kubernetes.io/projected/0fd70b58-6f75-415c-b26e-303faf172e6c-kube-api-access-89lmc\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:19.131099 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.130862 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fd70b58-6f75-415c-b26e-303faf172e6c-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:19.131099 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.130870 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fd70b58-6f75-415c-b26e-303faf172e6c-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:19.220905 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.220870 5615 generic.go:356] "Generic (PLEG): container finished" podID="0fd70b58-6f75-415c-b26e-303faf172e6c" containerID="136f7114eef954168a0ef368c9a05ca5f7c7659623bb98a30aaa963bc13d1777" exitCode=0 Nov 03 22:08:19.221054 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.220923 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" Nov 03 22:08:19.221054 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.220939 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" event={"ID":"0fd70b58-6f75-415c-b26e-303faf172e6c","Type":"ContainerDied","Data":"136f7114eef954168a0ef368c9a05ca5f7c7659623bb98a30aaa963bc13d1777"} Nov 03 22:08:19.221054 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.220981 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg" event={"ID":"0fd70b58-6f75-415c-b26e-303faf172e6c","Type":"ContainerDied","Data":"745517029e148e61125c37d9b12a11aca05882669e018998498a48414bfc5a4b"} Nov 03 22:08:19.221054 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.221002 5615 scope.go:122] "RemoveContainer" containerID="136f7114eef954168a0ef368c9a05ca5f7c7659623bb98a30aaa963bc13d1777" Nov 03 22:08:19.231310 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.231280 5615 scope.go:122] "RemoveContainer" containerID="8b98f3786960a3457eddfefdc91186507ff2528407e8354152811066dd3f544e" Nov 03 22:08:19.241286 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.241265 5615 scope.go:122] "RemoveContainer" containerID="136f7114eef954168a0ef368c9a05ca5f7c7659623bb98a30aaa963bc13d1777" Nov 03 22:08:19.241566 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:08:19.241531 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"136f7114eef954168a0ef368c9a05ca5f7c7659623bb98a30aaa963bc13d1777\": container with ID starting with 136f7114eef954168a0ef368c9a05ca5f7c7659623bb98a30aaa963bc13d1777 not found: ID does not exist" containerID="136f7114eef954168a0ef368c9a05ca5f7c7659623bb98a30aaa963bc13d1777" Nov 03 22:08:19.241660 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.241577 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"136f7114eef954168a0ef368c9a05ca5f7c7659623bb98a30aaa963bc13d1777"} err="failed to get container status \"136f7114eef954168a0ef368c9a05ca5f7c7659623bb98a30aaa963bc13d1777\": rpc error: code = NotFound desc = could not find container \"136f7114eef954168a0ef368c9a05ca5f7c7659623bb98a30aaa963bc13d1777\": container with ID starting with 136f7114eef954168a0ef368c9a05ca5f7c7659623bb98a30aaa963bc13d1777 not found: ID does not exist" Nov 03 22:08:19.241660 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.241598 5615 scope.go:122] "RemoveContainer" containerID="8b98f3786960a3457eddfefdc91186507ff2528407e8354152811066dd3f544e" Nov 03 22:08:19.241837 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:08:19.241815 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b98f3786960a3457eddfefdc91186507ff2528407e8354152811066dd3f544e\": container with ID starting with 8b98f3786960a3457eddfefdc91186507ff2528407e8354152811066dd3f544e not found: ID does not exist" containerID="8b98f3786960a3457eddfefdc91186507ff2528407e8354152811066dd3f544e" Nov 03 22:08:19.241884 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.241845 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b98f3786960a3457eddfefdc91186507ff2528407e8354152811066dd3f544e"} err="failed to get container status \"8b98f3786960a3457eddfefdc91186507ff2528407e8354152811066dd3f544e\": rpc error: code = NotFound desc = could not find container \"8b98f3786960a3457eddfefdc91186507ff2528407e8354152811066dd3f544e\": container with ID starting with 8b98f3786960a3457eddfefdc91186507ff2528407e8354152811066dd3f544e not found: ID does not exist" Nov 03 22:08:19.390466 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.390360 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg"] Nov 03 22:08:19.396457 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.396428 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a6219b-openshift-default-748487bf75-h92bg"] Nov 03 22:08:19.650711 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:19.650629 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fd70b58-6f75-415c-b26e-303faf172e6c" path="/var/lib/kubelet/pods/0fd70b58-6f75-415c-b26e-303faf172e6c/volumes" Nov 03 22:08:23.683522 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:23.683460 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 22:08:23.776955 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:23.776914 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istio-data\") pod \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " Nov 03 22:08:23.777158 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:23.776967 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-workload-certs\") pod \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " Nov 03 22:08:23.777158 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:23.777041 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istio-podinfo\") pod \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " Nov 03 22:08:23.777158 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:23.777119 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istiod-ca-cert\") pod \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " Nov 03 22:08:23.777158 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:23.777146 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-kube-api-access-xgjwk\" (UniqueName: \"kubernetes.io/projected/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-kube-api-access-xgjwk\") pod \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " Nov 03 22:08:23.777314 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:23.777174 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-workload-socket\") pod \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " Nov 03 22:08:23.777314 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:23.777211 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istio-token\" (UniqueName: \"kubernetes.io/projected/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istio-token\") pod \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " Nov 03 22:08:23.777314 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:23.777229 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istio-data" pod "a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b" (UID: "a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:23.777314 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:23.777239 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istio-envoy\") pod \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " Nov 03 22:08:23.777314 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:23.777267 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-workload-certs" pod "a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b" (UID: "a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:23.777314 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:23.777279 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-credential-socket\") pod \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\" (UID: \"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b\") " Nov 03 22:08:23.777759 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:23.777733 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:23.777854 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:23.777762 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:23.777979 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:23.777911 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-workload-socket" pod "a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b" (UID: "a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:23.778137 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:23.778010 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-credential-socket" pod "a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b" (UID: "a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:23.778137 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:23.778051 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istiod-ca-cert" pod "a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b" (UID: "a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:08:23.779645 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:23.779623 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istio-podinfo" pod "a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b" (UID: "a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:08:23.779752 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:23.779686 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istio-token" pod "a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b" (UID: "a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:23.779849 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:23.779832 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istio-envoy" pod "a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b" (UID: "a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:23.779897 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:23.779863 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-kube-api-access-xgjwk" pod "a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b" (UID: "a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b"). InnerVolumeSpecName "kube-api-access-xgjwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:23.879030 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:23.878984 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:23.879030 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:23.879022 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:23.879030 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:23.879035 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgjwk\" (UniqueName: \"kubernetes.io/projected/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-kube-api-access-xgjwk\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:23.879300 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:23.879051 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:23.879300 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:23.879063 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:23.879300 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:23.879105 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:23.879300 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:23.879119 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:24.253859 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:24.253823 5615 generic.go:356] "Generic (PLEG): container finished" podID="a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b" containerID="1e92f3208be893b817dd50dbc19bc41713288b24dbb2c2a80fd7c5a495e63e94" exitCode=0 Nov 03 22:08:24.254062 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:24.253872 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" Nov 03 22:08:24.254062 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:24.253910 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" event={"ID":"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b","Type":"ContainerDied","Data":"1e92f3208be893b817dd50dbc19bc41713288b24dbb2c2a80fd7c5a495e63e94"} Nov 03 22:08:24.254062 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:24.253969 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t" event={"ID":"a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b","Type":"ContainerDied","Data":"9d03e5156d564f87d89c147d1aa62eeb42714eabcd5c33e39229e37e7601efa8"} Nov 03 22:08:24.254062 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:24.253994 5615 scope.go:122] "RemoveContainer" containerID="1e92f3208be893b817dd50dbc19bc41713288b24dbb2c2a80fd7c5a495e63e94" Nov 03 22:08:24.264540 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:24.264521 5615 scope.go:122] "RemoveContainer" containerID="1e92f3208be893b817dd50dbc19bc41713288b24dbb2c2a80fd7c5a495e63e94" Nov 03 22:08:24.264833 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:08:24.264814 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e92f3208be893b817dd50dbc19bc41713288b24dbb2c2a80fd7c5a495e63e94\": container with ID starting with 1e92f3208be893b817dd50dbc19bc41713288b24dbb2c2a80fd7c5a495e63e94 not found: ID does not exist" containerID="1e92f3208be893b817dd50dbc19bc41713288b24dbb2c2a80fd7c5a495e63e94" Nov 03 22:08:24.264889 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:24.264842 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e92f3208be893b817dd50dbc19bc41713288b24dbb2c2a80fd7c5a495e63e94"} err="failed to get container status \"1e92f3208be893b817dd50dbc19bc41713288b24dbb2c2a80fd7c5a495e63e94\": rpc error: code = NotFound desc = could not find container \"1e92f3208be893b817dd50dbc19bc41713288b24dbb2c2a80fd7c5a495e63e94\": container with ID starting with 1e92f3208be893b817dd50dbc19bc41713288b24dbb2c2a80fd7c5a495e63e94 not found: ID does not exist" Nov 03 22:08:24.423488 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:24.423449 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t"] Nov 03 22:08:24.427347 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:24.427323 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-pd24t"] Nov 03 22:08:25.650270 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:25.650236 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b" path="/var/lib/kubelet/pods/a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b/volumes" Nov 03 22:08:33.830166 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:33.830125 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8"] Nov 03 22:08:33.830686 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:33.830440 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8" podUID="ef206c3a-762d-450d-9fb5-5a537392bffc" containerName="maas-api" containerID="cri-o://ce4d1e37084c7a1448bda70c49ffa2b9c562346a3f5b69a0fe2298b4a51f7fa7" gracePeriod=30 Nov 03 22:08:34.097321 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:34.097263 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8" Nov 03 22:08:34.186726 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:34.186236 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef206c3a-762d-450d-9fb5-5a537392bffc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef206c3a-762d-450d-9fb5-5a537392bffc-metrics-certs\") pod \"ef206c3a-762d-450d-9fb5-5a537392bffc\" (UID: \"ef206c3a-762d-450d-9fb5-5a537392bffc\") " Nov 03 22:08:34.186726 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:34.186321 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef206c3a-762d-450d-9fb5-5a537392bffc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef206c3a-762d-450d-9fb5-5a537392bffc-openshift-service-ca\") pod \"ef206c3a-762d-450d-9fb5-5a537392bffc\" (UID: \"ef206c3a-762d-450d-9fb5-5a537392bffc\") " Nov 03 22:08:34.186726 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:34.186380 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef206c3a-762d-450d-9fb5-5a537392bffc-kube-api-access-9rr46\" (UniqueName: \"kubernetes.io/projected/ef206c3a-762d-450d-9fb5-5a537392bffc-kube-api-access-9rr46\") pod \"ef206c3a-762d-450d-9fb5-5a537392bffc\" (UID: \"ef206c3a-762d-450d-9fb5-5a537392bffc\") " Nov 03 22:08:34.186726 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:34.186412 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef206c3a-762d-450d-9fb5-5a537392bffc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef206c3a-762d-450d-9fb5-5a537392bffc-maas-api-tls\") pod \"ef206c3a-762d-450d-9fb5-5a537392bffc\" (UID: \"ef206c3a-762d-450d-9fb5-5a537392bffc\") " Nov 03 22:08:34.187310 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:34.187276 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef206c3a-762d-450d-9fb5-5a537392bffc-openshift-service-ca" pod "ef206c3a-762d-450d-9fb5-5a537392bffc" (UID: "ef206c3a-762d-450d-9fb5-5a537392bffc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:08:34.189760 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:34.189723 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef206c3a-762d-450d-9fb5-5a537392bffc-maas-api-tls" pod "ef206c3a-762d-450d-9fb5-5a537392bffc" (UID: "ef206c3a-762d-450d-9fb5-5a537392bffc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:08:34.189976 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:34.189949 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef206c3a-762d-450d-9fb5-5a537392bffc-metrics-certs" pod "ef206c3a-762d-450d-9fb5-5a537392bffc" (UID: "ef206c3a-762d-450d-9fb5-5a537392bffc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:08:34.190276 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:34.190256 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef206c3a-762d-450d-9fb5-5a537392bffc-kube-api-access-9rr46" pod "ef206c3a-762d-450d-9fb5-5a537392bffc" (UID: "ef206c3a-762d-450d-9fb5-5a537392bffc"). InnerVolumeSpecName "kube-api-access-9rr46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:34.290417 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:34.289330 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef206c3a-762d-450d-9fb5-5a537392bffc-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:34.290417 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:34.289377 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rr46\" (UniqueName: \"kubernetes.io/projected/ef206c3a-762d-450d-9fb5-5a537392bffc-kube-api-access-9rr46\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:34.290417 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:34.289396 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef206c3a-762d-450d-9fb5-5a537392bffc-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:34.290417 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:34.289410 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef206c3a-762d-450d-9fb5-5a537392bffc-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:34.306398 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:34.306359 5615 generic.go:356] "Generic (PLEG): container finished" podID="ef206c3a-762d-450d-9fb5-5a537392bffc" containerID="ce4d1e37084c7a1448bda70c49ffa2b9c562346a3f5b69a0fe2298b4a51f7fa7" exitCode=0 Nov 03 22:08:34.306588 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:34.306428 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8" Nov 03 22:08:34.306588 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:34.306456 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8" event={"ID":"ef206c3a-762d-450d-9fb5-5a537392bffc","Type":"ContainerDied","Data":"ce4d1e37084c7a1448bda70c49ffa2b9c562346a3f5b69a0fe2298b4a51f7fa7"} Nov 03 22:08:34.306588 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:34.306538 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8" event={"ID":"ef206c3a-762d-450d-9fb5-5a537392bffc","Type":"ContainerDied","Data":"e56a5aae820478918031e8d96349757cbbb0885717d0d498d39c67eb8be284ac"} Nov 03 22:08:34.306588 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:34.306561 5615 scope.go:122] "RemoveContainer" containerID="ce4d1e37084c7a1448bda70c49ffa2b9c562346a3f5b69a0fe2298b4a51f7fa7" Nov 03 22:08:34.315927 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:34.315903 5615 scope.go:122] "RemoveContainer" containerID="ce4d1e37084c7a1448bda70c49ffa2b9c562346a3f5b69a0fe2298b4a51f7fa7" Nov 03 22:08:34.316202 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:08:34.316184 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce4d1e37084c7a1448bda70c49ffa2b9c562346a3f5b69a0fe2298b4a51f7fa7\": container with ID starting with ce4d1e37084c7a1448bda70c49ffa2b9c562346a3f5b69a0fe2298b4a51f7fa7 not found: ID does not exist" containerID="ce4d1e37084c7a1448bda70c49ffa2b9c562346a3f5b69a0fe2298b4a51f7fa7" Nov 03 22:08:34.316257 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:34.316209 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce4d1e37084c7a1448bda70c49ffa2b9c562346a3f5b69a0fe2298b4a51f7fa7"} err="failed to get container status \"ce4d1e37084c7a1448bda70c49ffa2b9c562346a3f5b69a0fe2298b4a51f7fa7\": rpc error: code = NotFound desc = could not find container \"ce4d1e37084c7a1448bda70c49ffa2b9c562346a3f5b69a0fe2298b4a51f7fa7\": container with ID starting with ce4d1e37084c7a1448bda70c49ffa2b9c562346a3f5b69a0fe2298b4a51f7fa7 not found: ID does not exist" Nov 03 22:08:34.474817 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:34.474731 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8"] Nov 03 22:08:34.479843 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:34.479812 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c70fca-744c4686-pqjb8"] Nov 03 22:08:35.652156 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:35.652123 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef206c3a-762d-450d-9fb5-5a537392bffc" path="/var/lib/kubelet/pods/ef206c3a-762d-450d-9fb5-5a537392bffc/volumes" Nov 03 22:08:40.639980 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:08:40.639939 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:08:41.049277 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:41.049234 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx"] Nov 03 22:08:41.049559 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:41.049497 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" podUID="782d9d6a-583d-4bdc-af1e-d87d81db3d4a" containerName="istio-proxy" containerID="cri-o://b34cd6395e5acac9322bb639c7107b94cc6ad056e87677c505e4ac5859799e93" gracePeriod=30 Nov 03 22:08:41.560982 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:41.560948 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj"] Nov 03 22:08:41.561224 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:41.561200 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" podUID="7803de3d-9e9c-475a-9227-74535c9c1a68" containerName="istio-proxy" containerID="cri-o://6329bddbbeea5304c526adfd1a606ee5281f84a849bfa800f9507d1cd87d377d" gracePeriod=30 Nov 03 22:08:44.214599 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:44.214565 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-776jj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:08:44.215034 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:44.214620 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" podUID="7803de3d-9e9c-475a-9227-74535c9c1a68" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:08:46.300870 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.300841 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 22:08:46.370423 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.370339 5615 generic.go:356] "Generic (PLEG): container finished" podID="782d9d6a-583d-4bdc-af1e-d87d81db3d4a" containerID="b34cd6395e5acac9322bb639c7107b94cc6ad056e87677c505e4ac5859799e93" exitCode=0 Nov 03 22:08:46.370423 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.370387 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" Nov 03 22:08:46.370613 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.370423 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" event={"ID":"782d9d6a-583d-4bdc-af1e-d87d81db3d4a","Type":"ContainerDied","Data":"b34cd6395e5acac9322bb639c7107b94cc6ad056e87677c505e4ac5859799e93"} Nov 03 22:08:46.370613 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.370476 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx" event={"ID":"782d9d6a-583d-4bdc-af1e-d87d81db3d4a","Type":"ContainerDied","Data":"d60d45f79953c6eec7ee0e3f77d91aed1ba8756d1d4b469f721d1bac15aadf35"} Nov 03 22:08:46.370613 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.370502 5615 scope.go:122] "RemoveContainer" containerID="b34cd6395e5acac9322bb639c7107b94cc6ad056e87677c505e4ac5859799e93" Nov 03 22:08:46.379312 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.379291 5615 scope.go:122] "RemoveContainer" containerID="aa5d5e70a16b8e2a7c9bc5b90e58a014962e173d9931e48f17f47cd709d08ca2" Nov 03 22:08:46.387445 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.387430 5615 scope.go:122] "RemoveContainer" containerID="b34cd6395e5acac9322bb639c7107b94cc6ad056e87677c505e4ac5859799e93" Nov 03 22:08:46.387714 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:08:46.387696 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b34cd6395e5acac9322bb639c7107b94cc6ad056e87677c505e4ac5859799e93\": container with ID starting with b34cd6395e5acac9322bb639c7107b94cc6ad056e87677c505e4ac5859799e93 not found: ID does not exist" containerID="b34cd6395e5acac9322bb639c7107b94cc6ad056e87677c505e4ac5859799e93" Nov 03 22:08:46.387791 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.387722 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b34cd6395e5acac9322bb639c7107b94cc6ad056e87677c505e4ac5859799e93"} err="failed to get container status \"b34cd6395e5acac9322bb639c7107b94cc6ad056e87677c505e4ac5859799e93\": rpc error: code = NotFound desc = could not find container \"b34cd6395e5acac9322bb639c7107b94cc6ad056e87677c505e4ac5859799e93\": container with ID starting with b34cd6395e5acac9322bb639c7107b94cc6ad056e87677c505e4ac5859799e93 not found: ID does not exist" Nov 03 22:08:46.387791 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.387742 5615 scope.go:122] "RemoveContainer" containerID="aa5d5e70a16b8e2a7c9bc5b90e58a014962e173d9931e48f17f47cd709d08ca2" Nov 03 22:08:46.388002 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:08:46.387984 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa5d5e70a16b8e2a7c9bc5b90e58a014962e173d9931e48f17f47cd709d08ca2\": container with ID starting with aa5d5e70a16b8e2a7c9bc5b90e58a014962e173d9931e48f17f47cd709d08ca2 not found: ID does not exist" containerID="aa5d5e70a16b8e2a7c9bc5b90e58a014962e173d9931e48f17f47cd709d08ca2" Nov 03 22:08:46.388043 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.388005 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa5d5e70a16b8e2a7c9bc5b90e58a014962e173d9931e48f17f47cd709d08ca2"} err="failed to get container status \"aa5d5e70a16b8e2a7c9bc5b90e58a014962e173d9931e48f17f47cd709d08ca2\": rpc error: code = NotFound desc = could not find container \"aa5d5e70a16b8e2a7c9bc5b90e58a014962e173d9931e48f17f47cd709d08ca2\": container with ID starting with aa5d5e70a16b8e2a7c9bc5b90e58a014962e173d9931e48f17f47cd709d08ca2 not found: ID does not exist" Nov 03 22:08:46.420988 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.420952 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-kube-api-access-kgxsh\" (UniqueName: \"kubernetes.io/projected/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-kube-api-access-kgxsh\") pod \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " Nov 03 22:08:46.420988 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.420993 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istiod-ca-cert\") pod \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " Nov 03 22:08:46.421223 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.421034 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istio-data\") pod \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " Nov 03 22:08:46.421223 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.421122 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istio-podinfo\") pod \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " Nov 03 22:08:46.421223 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.421150 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-workload-certs\") pod \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " Nov 03 22:08:46.421385 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.421285 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-workload-socket\") pod \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " Nov 03 22:08:46.421385 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.421359 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istio-envoy\") pod \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " Nov 03 22:08:46.421504 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.421390 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-credential-socket\") pod \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " Nov 03 22:08:46.421504 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.421422 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istio-data" pod "782d9d6a-583d-4bdc-af1e-d87d81db3d4a" (UID: "782d9d6a-583d-4bdc-af1e-d87d81db3d4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:46.421504 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.421433 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istio-token\" (UniqueName: \"kubernetes.io/projected/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istio-token\") pod \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\" (UID: \"782d9d6a-583d-4bdc-af1e-d87d81db3d4a\") " Nov 03 22:08:46.421504 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.421465 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-workload-certs" pod "782d9d6a-583d-4bdc-af1e-d87d81db3d4a" (UID: "782d9d6a-583d-4bdc-af1e-d87d81db3d4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:46.421738 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.421514 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-workload-socket" pod "782d9d6a-583d-4bdc-af1e-d87d81db3d4a" (UID: "782d9d6a-583d-4bdc-af1e-d87d81db3d4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:46.421738 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.421530 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istiod-ca-cert" pod "782d9d6a-583d-4bdc-af1e-d87d81db3d4a" (UID: "782d9d6a-583d-4bdc-af1e-d87d81db3d4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:08:46.421738 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.421697 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-credential-socket" pod "782d9d6a-583d-4bdc-af1e-d87d81db3d4a" (UID: "782d9d6a-583d-4bdc-af1e-d87d81db3d4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:46.421906 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.421850 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:46.421906 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.421878 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:46.421906 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.421895 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:46.422023 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.421909 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:46.422023 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.421923 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:46.423438 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.423415 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-kube-api-access-kgxsh" pod "782d9d6a-583d-4bdc-af1e-d87d81db3d4a" (UID: "782d9d6a-583d-4bdc-af1e-d87d81db3d4a"). InnerVolumeSpecName "kube-api-access-kgxsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:46.423551 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.423529 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istio-envoy" pod "782d9d6a-583d-4bdc-af1e-d87d81db3d4a" (UID: "782d9d6a-583d-4bdc-af1e-d87d81db3d4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:46.423901 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.423882 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istio-podinfo" pod "782d9d6a-583d-4bdc-af1e-d87d81db3d4a" (UID: "782d9d6a-583d-4bdc-af1e-d87d81db3d4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:08:46.423947 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.423902 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istio-token" pod "782d9d6a-583d-4bdc-af1e-d87d81db3d4a" (UID: "782d9d6a-583d-4bdc-af1e-d87d81db3d4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:46.522572 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.522542 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:46.522572 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.522569 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:46.522792 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.522580 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:46.522792 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.522590 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgxsh\" (UniqueName: \"kubernetes.io/projected/782d9d6a-583d-4bdc-af1e-d87d81db3d4a-kube-api-access-kgxsh\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:46.793958 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.793924 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx"] Nov 03 22:08:46.798616 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.798583 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c70fca-openshift-default-57d4dbbf59-cc6dx"] Nov 03 22:08:46.799477 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.799453 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 22:08:46.926187 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.926152 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7803de3d-9e9c-475a-9227-74535c9c1a68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7803de3d-9e9c-475a-9227-74535c9c1a68-istio-podinfo\") pod \"7803de3d-9e9c-475a-9227-74535c9c1a68\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " Nov 03 22:08:46.926378 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.926198 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-credential-socket\") pod \"7803de3d-9e9c-475a-9227-74535c9c1a68\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " Nov 03 22:08:46.926378 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.926228 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7803de3d-9e9c-475a-9227-74535c9c1a68-kube-api-access-6xmjc\" (UniqueName: \"kubernetes.io/projected/7803de3d-9e9c-475a-9227-74535c9c1a68-kube-api-access-6xmjc\") pod \"7803de3d-9e9c-475a-9227-74535c9c1a68\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " Nov 03 22:08:46.926378 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.926299 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-istio-data\") pod \"7803de3d-9e9c-475a-9227-74535c9c1a68\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " Nov 03 22:08:46.926378 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.926333 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-istio-envoy\") pod \"7803de3d-9e9c-475a-9227-74535c9c1a68\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " Nov 03 22:08:46.926631 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.926399 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7803de3d-9e9c-475a-9227-74535c9c1a68-istio-token\" (UniqueName: \"kubernetes.io/projected/7803de3d-9e9c-475a-9227-74535c9c1a68-istio-token\") pod \"7803de3d-9e9c-475a-9227-74535c9c1a68\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " Nov 03 22:08:46.926631 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.926492 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-workload-socket\") pod \"7803de3d-9e9c-475a-9227-74535c9c1a68\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " Nov 03 22:08:46.926631 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.926536 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-workload-certs\") pod \"7803de3d-9e9c-475a-9227-74535c9c1a68\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " Nov 03 22:08:46.926631 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.926571 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7803de3d-9e9c-475a-9227-74535c9c1a68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7803de3d-9e9c-475a-9227-74535c9c1a68-istiod-ca-cert\") pod \"7803de3d-9e9c-475a-9227-74535c9c1a68\" (UID: \"7803de3d-9e9c-475a-9227-74535c9c1a68\") " Nov 03 22:08:46.926849 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.926665 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-istio-data" pod "7803de3d-9e9c-475a-9227-74535c9c1a68" (UID: "7803de3d-9e9c-475a-9227-74535c9c1a68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:46.926849 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.926808 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-credential-socket" pod "7803de3d-9e9c-475a-9227-74535c9c1a68" (UID: "7803de3d-9e9c-475a-9227-74535c9c1a68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:46.926962 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.926839 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-workload-socket" pod "7803de3d-9e9c-475a-9227-74535c9c1a68" (UID: "7803de3d-9e9c-475a-9227-74535c9c1a68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:46.927019 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.926965 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7803de3d-9e9c-475a-9227-74535c9c1a68-istiod-ca-cert" pod "7803de3d-9e9c-475a-9227-74535c9c1a68" (UID: "7803de3d-9e9c-475a-9227-74535c9c1a68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:08:46.927098 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.927019 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:46.927098 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.927053 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:46.927215 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.927069 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:46.927267 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.927225 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-workload-certs" pod "7803de3d-9e9c-475a-9227-74535c9c1a68" (UID: "7803de3d-9e9c-475a-9227-74535c9c1a68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:46.929151 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.929064 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-istio-envoy" pod "7803de3d-9e9c-475a-9227-74535c9c1a68" (UID: "7803de3d-9e9c-475a-9227-74535c9c1a68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:46.929408 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.929388 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7803de3d-9e9c-475a-9227-74535c9c1a68-kube-api-access-6xmjc" pod "7803de3d-9e9c-475a-9227-74535c9c1a68" (UID: "7803de3d-9e9c-475a-9227-74535c9c1a68"). InnerVolumeSpecName "kube-api-access-6xmjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:46.929506 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.929412 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7803de3d-9e9c-475a-9227-74535c9c1a68-istio-podinfo" pod "7803de3d-9e9c-475a-9227-74535c9c1a68" (UID: "7803de3d-9e9c-475a-9227-74535c9c1a68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:08:46.929506 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:46.929420 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7803de3d-9e9c-475a-9227-74535c9c1a68-istio-token" pod "7803de3d-9e9c-475a-9227-74535c9c1a68" (UID: "7803de3d-9e9c-475a-9227-74535c9c1a68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:47.028188 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:47.028149 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xmjc\" (UniqueName: \"kubernetes.io/projected/7803de3d-9e9c-475a-9227-74535c9c1a68-kube-api-access-6xmjc\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:47.028188 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:47.028180 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:47.028188 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:47.028190 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7803de3d-9e9c-475a-9227-74535c9c1a68-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:47.028188 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:47.028199 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7803de3d-9e9c-475a-9227-74535c9c1a68-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:47.028568 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:47.028207 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7803de3d-9e9c-475a-9227-74535c9c1a68-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:47.028568 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:47.028216 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7803de3d-9e9c-475a-9227-74535c9c1a68-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:47.377736 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:47.377694 5615 generic.go:356] "Generic (PLEG): container finished" podID="7803de3d-9e9c-475a-9227-74535c9c1a68" containerID="6329bddbbeea5304c526adfd1a606ee5281f84a849bfa800f9507d1cd87d377d" exitCode=0 Nov 03 22:08:47.378204 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:47.377742 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" Nov 03 22:08:47.378204 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:47.377752 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" event={"ID":"7803de3d-9e9c-475a-9227-74535c9c1a68","Type":"ContainerDied","Data":"6329bddbbeea5304c526adfd1a606ee5281f84a849bfa800f9507d1cd87d377d"} Nov 03 22:08:47.378204 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:47.377802 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj" event={"ID":"7803de3d-9e9c-475a-9227-74535c9c1a68","Type":"ContainerDied","Data":"3b84bbc316d370d1da70b100413a3f8cfc99fefc6749c42e472999c0adc58313"} Nov 03 22:08:47.378204 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:47.377825 5615 scope.go:122] "RemoveContainer" containerID="6329bddbbeea5304c526adfd1a606ee5281f84a849bfa800f9507d1cd87d377d" Nov 03 22:08:47.387416 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:47.387397 5615 scope.go:122] "RemoveContainer" containerID="6329bddbbeea5304c526adfd1a606ee5281f84a849bfa800f9507d1cd87d377d" Nov 03 22:08:47.387684 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:08:47.387661 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6329bddbbeea5304c526adfd1a606ee5281f84a849bfa800f9507d1cd87d377d\": container with ID starting with 6329bddbbeea5304c526adfd1a606ee5281f84a849bfa800f9507d1cd87d377d not found: ID does not exist" containerID="6329bddbbeea5304c526adfd1a606ee5281f84a849bfa800f9507d1cd87d377d" Nov 03 22:08:47.387756 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:47.387697 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6329bddbbeea5304c526adfd1a606ee5281f84a849bfa800f9507d1cd87d377d"} err="failed to get container status \"6329bddbbeea5304c526adfd1a606ee5281f84a849bfa800f9507d1cd87d377d\": rpc error: code = NotFound desc = could not find container \"6329bddbbeea5304c526adfd1a606ee5281f84a849bfa800f9507d1cd87d377d\": container with ID starting with 6329bddbbeea5304c526adfd1a606ee5281f84a849bfa800f9507d1cd87d377d not found: ID does not exist" Nov 03 22:08:47.545891 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:47.545856 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj"] Nov 03 22:08:47.557293 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:47.557264 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-776jj"] Nov 03 22:08:47.650624 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:47.650549 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7803de3d-9e9c-475a-9227-74535c9c1a68" path="/var/lib/kubelet/pods/7803de3d-9e9c-475a-9227-74535c9c1a68/volumes" Nov 03 22:08:47.651024 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:47.651010 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="782d9d6a-583d-4bdc-af1e-d87d81db3d4a" path="/var/lib/kubelet/pods/782d9d6a-583d-4bdc-af1e-d87d81db3d4a/volumes" Nov 03 22:08:49.477874 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:49.477836 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt"] Nov 03 22:08:49.478364 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:49.478039 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt" podUID="b037fb5a-8768-4e13-b190-b5226818fd81" containerName="maas-api" containerID="cri-o://86e4d83bf830865c8b7d34e058e47f23a3148758b4526ba9b9fbaf7850340a03" gracePeriod=30 Nov 03 22:08:49.738380 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:49.738345 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt" Nov 03 22:08:49.858184 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:49.858149 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b037fb5a-8768-4e13-b190-b5226818fd81-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b037fb5a-8768-4e13-b190-b5226818fd81-metrics-certs\") pod \"b037fb5a-8768-4e13-b190-b5226818fd81\" (UID: \"b037fb5a-8768-4e13-b190-b5226818fd81\") " Nov 03 22:08:49.858362 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:49.858216 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b037fb5a-8768-4e13-b190-b5226818fd81-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b037fb5a-8768-4e13-b190-b5226818fd81-maas-api-tls\") pod \"b037fb5a-8768-4e13-b190-b5226818fd81\" (UID: \"b037fb5a-8768-4e13-b190-b5226818fd81\") " Nov 03 22:08:49.858362 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:49.858286 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b037fb5a-8768-4e13-b190-b5226818fd81-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b037fb5a-8768-4e13-b190-b5226818fd81-openshift-service-ca\") pod \"b037fb5a-8768-4e13-b190-b5226818fd81\" (UID: \"b037fb5a-8768-4e13-b190-b5226818fd81\") " Nov 03 22:08:49.858362 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:49.858355 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b037fb5a-8768-4e13-b190-b5226818fd81-kube-api-access-nb26n\" (UniqueName: \"kubernetes.io/projected/b037fb5a-8768-4e13-b190-b5226818fd81-kube-api-access-nb26n\") pod \"b037fb5a-8768-4e13-b190-b5226818fd81\" (UID: \"b037fb5a-8768-4e13-b190-b5226818fd81\") " Nov 03 22:08:49.858687 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:49.858660 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b037fb5a-8768-4e13-b190-b5226818fd81-openshift-service-ca" pod "b037fb5a-8768-4e13-b190-b5226818fd81" (UID: "b037fb5a-8768-4e13-b190-b5226818fd81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:08:49.860781 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:49.860756 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b037fb5a-8768-4e13-b190-b5226818fd81-maas-api-tls" pod "b037fb5a-8768-4e13-b190-b5226818fd81" (UID: "b037fb5a-8768-4e13-b190-b5226818fd81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:08:49.861481 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:49.861459 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b037fb5a-8768-4e13-b190-b5226818fd81-metrics-certs" pod "b037fb5a-8768-4e13-b190-b5226818fd81" (UID: "b037fb5a-8768-4e13-b190-b5226818fd81"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:08:49.861590 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:49.861500 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b037fb5a-8768-4e13-b190-b5226818fd81-kube-api-access-nb26n" pod "b037fb5a-8768-4e13-b190-b5226818fd81" (UID: "b037fb5a-8768-4e13-b190-b5226818fd81"). InnerVolumeSpecName "kube-api-access-nb26n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:49.959654 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:49.959613 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b037fb5a-8768-4e13-b190-b5226818fd81-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:49.959654 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:49.959647 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b037fb5a-8768-4e13-b190-b5226818fd81-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:49.959654 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:49.959659 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b037fb5a-8768-4e13-b190-b5226818fd81-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:49.959906 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:49.959670 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb26n\" (UniqueName: \"kubernetes.io/projected/b037fb5a-8768-4e13-b190-b5226818fd81-kube-api-access-nb26n\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:50.394232 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:50.394196 5615 generic.go:356] "Generic (PLEG): container finished" podID="b037fb5a-8768-4e13-b190-b5226818fd81" containerID="86e4d83bf830865c8b7d34e058e47f23a3148758b4526ba9b9fbaf7850340a03" exitCode=0 Nov 03 22:08:50.394425 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:50.394245 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt" Nov 03 22:08:50.394425 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:50.394288 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt" event={"ID":"b037fb5a-8768-4e13-b190-b5226818fd81","Type":"ContainerDied","Data":"86e4d83bf830865c8b7d34e058e47f23a3148758b4526ba9b9fbaf7850340a03"} Nov 03 22:08:50.394425 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:50.394348 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt" event={"ID":"b037fb5a-8768-4e13-b190-b5226818fd81","Type":"ContainerDied","Data":"7da20b6dbcdae452eb3333632e233fdf547587bf2ca8aa274006714f5298639f"} Nov 03 22:08:50.394425 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:50.394373 5615 scope.go:122] "RemoveContainer" containerID="86e4d83bf830865c8b7d34e058e47f23a3148758b4526ba9b9fbaf7850340a03" Nov 03 22:08:50.404156 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:50.404129 5615 scope.go:122] "RemoveContainer" containerID="86e4d83bf830865c8b7d34e058e47f23a3148758b4526ba9b9fbaf7850340a03" Nov 03 22:08:50.404429 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:08:50.404402 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86e4d83bf830865c8b7d34e058e47f23a3148758b4526ba9b9fbaf7850340a03\": container with ID starting with 86e4d83bf830865c8b7d34e058e47f23a3148758b4526ba9b9fbaf7850340a03 not found: ID does not exist" containerID="86e4d83bf830865c8b7d34e058e47f23a3148758b4526ba9b9fbaf7850340a03" Nov 03 22:08:50.404540 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:50.404439 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86e4d83bf830865c8b7d34e058e47f23a3148758b4526ba9b9fbaf7850340a03"} err="failed to get container status \"86e4d83bf830865c8b7d34e058e47f23a3148758b4526ba9b9fbaf7850340a03\": rpc error: code = NotFound desc = could not find container \"86e4d83bf830865c8b7d34e058e47f23a3148758b4526ba9b9fbaf7850340a03\": container with ID starting with 86e4d83bf830865c8b7d34e058e47f23a3148758b4526ba9b9fbaf7850340a03 not found: ID does not exist" Nov 03 22:08:50.563009 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:50.562966 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt"] Nov 03 22:08:50.569057 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:50.569024 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-099a5e-7f485c4c9d-g4jjt"] Nov 03 22:08:51.650900 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:51.650865 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b037fb5a-8768-4e13-b190-b5226818fd81" path="/var/lib/kubelet/pods/b037fb5a-8768-4e13-b190-b5226818fd81/volumes" Nov 03 22:08:55.230135 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:55.230093 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb"] Nov 03 22:08:55.230591 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:55.230290 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb" podUID="9592ace4-6808-4b59-8bbc-58243dc78e12" containerName="maas-api" containerID="cri-o://ed6e5af3628f4c2ce87875ed85474f9af2d7dea14519c27483f324652ea6124f" gracePeriod=30 Nov 03 22:08:55.427982 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:55.427947 5615 generic.go:356] "Generic (PLEG): container finished" podID="9592ace4-6808-4b59-8bbc-58243dc78e12" containerID="ed6e5af3628f4c2ce87875ed85474f9af2d7dea14519c27483f324652ea6124f" exitCode=0 Nov 03 22:08:55.428193 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:55.428023 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb" event={"ID":"9592ace4-6808-4b59-8bbc-58243dc78e12","Type":"ContainerDied","Data":"ed6e5af3628f4c2ce87875ed85474f9af2d7dea14519c27483f324652ea6124f"} Nov 03 22:08:55.490130 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:55.490064 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb" Nov 03 22:08:55.513359 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:55.513327 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9592ace4-6808-4b59-8bbc-58243dc78e12-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9592ace4-6808-4b59-8bbc-58243dc78e12-openshift-service-ca\") pod \"9592ace4-6808-4b59-8bbc-58243dc78e12\" (UID: \"9592ace4-6808-4b59-8bbc-58243dc78e12\") " Nov 03 22:08:55.513571 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:55.513386 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9592ace4-6808-4b59-8bbc-58243dc78e12-kube-api-access-knzqh\" (UniqueName: \"kubernetes.io/projected/9592ace4-6808-4b59-8bbc-58243dc78e12-kube-api-access-knzqh\") pod \"9592ace4-6808-4b59-8bbc-58243dc78e12\" (UID: \"9592ace4-6808-4b59-8bbc-58243dc78e12\") " Nov 03 22:08:55.513571 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:55.513431 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9592ace4-6808-4b59-8bbc-58243dc78e12-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9592ace4-6808-4b59-8bbc-58243dc78e12-maas-api-tls\") pod \"9592ace4-6808-4b59-8bbc-58243dc78e12\" (UID: \"9592ace4-6808-4b59-8bbc-58243dc78e12\") " Nov 03 22:08:55.513571 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:55.513476 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9592ace4-6808-4b59-8bbc-58243dc78e12-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9592ace4-6808-4b59-8bbc-58243dc78e12-metrics-certs\") pod \"9592ace4-6808-4b59-8bbc-58243dc78e12\" (UID: \"9592ace4-6808-4b59-8bbc-58243dc78e12\") " Nov 03 22:08:55.513778 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:55.513739 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9592ace4-6808-4b59-8bbc-58243dc78e12-openshift-service-ca" pod "9592ace4-6808-4b59-8bbc-58243dc78e12" (UID: "9592ace4-6808-4b59-8bbc-58243dc78e12"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:08:55.513866 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:55.513846 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9592ace4-6808-4b59-8bbc-58243dc78e12-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:55.515815 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:55.515784 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9592ace4-6808-4b59-8bbc-58243dc78e12-kube-api-access-knzqh" pod "9592ace4-6808-4b59-8bbc-58243dc78e12" (UID: "9592ace4-6808-4b59-8bbc-58243dc78e12"). InnerVolumeSpecName "kube-api-access-knzqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:55.515815 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:55.515801 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9592ace4-6808-4b59-8bbc-58243dc78e12-maas-api-tls" pod "9592ace4-6808-4b59-8bbc-58243dc78e12" (UID: "9592ace4-6808-4b59-8bbc-58243dc78e12"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:08:55.515815 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:55.515784 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9592ace4-6808-4b59-8bbc-58243dc78e12-metrics-certs" pod "9592ace4-6808-4b59-8bbc-58243dc78e12" (UID: "9592ace4-6808-4b59-8bbc-58243dc78e12"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:08:55.614726 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:55.614683 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9592ace4-6808-4b59-8bbc-58243dc78e12-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:55.614726 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:55.614721 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knzqh\" (UniqueName: \"kubernetes.io/projected/9592ace4-6808-4b59-8bbc-58243dc78e12-kube-api-access-knzqh\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:55.614937 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:55.614736 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9592ace4-6808-4b59-8bbc-58243dc78e12-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:56.434618 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:56.434580 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb" event={"ID":"9592ace4-6808-4b59-8bbc-58243dc78e12","Type":"ContainerDied","Data":"7113d5f3c5930224c385aa95ec7366fa17b1119e1065a3b4beea4df949875183"} Nov 03 22:08:56.434618 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:56.434611 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb" Nov 03 22:08:56.435271 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:56.434667 5615 scope.go:122] "RemoveContainer" containerID="ed6e5af3628f4c2ce87875ed85474f9af2d7dea14519c27483f324652ea6124f" Nov 03 22:08:56.601020 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:56.600979 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb"] Nov 03 22:08:56.652144 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:56.652111 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a1c634-54f4875f79-xjxvb"] Nov 03 22:08:57.650530 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:57.650498 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9592ace4-6808-4b59-8bbc-58243dc78e12" path="/var/lib/kubelet/pods/9592ace4-6808-4b59-8bbc-58243dc78e12/volumes" Nov 03 22:08:59.309004 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.308965 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2"] Nov 03 22:08:59.309444 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309411 5615 cpu_manager.go:414] "containerMap: removing container" podUID="0fd70b58-6f75-415c-b26e-303faf172e6c" containerName="istio-proxy" Nov 03 22:08:59.309444 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309423 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd70b58-6f75-415c-b26e-303faf172e6c" containerName="istio-proxy" Nov 03 22:08:59.309444 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309442 5615 cpu_manager.go:414] "containerMap: removing container" podUID="cb894762-26e0-4dd6-9e55-0a0aed4455c9" containerName="payload-pre-processing" Nov 03 22:08:59.309593 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309452 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb894762-26e0-4dd6-9e55-0a0aed4455c9" containerName="payload-pre-processing" Nov 03 22:08:59.309593 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309461 5615 cpu_manager.go:414] "containerMap: removing container" podUID="614c1da8-3cb5-4362-8774-b6b76ab79ac2" containerName="maas-api" Nov 03 22:08:59.309593 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309466 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="614c1da8-3cb5-4362-8774-b6b76ab79ac2" containerName="maas-api" Nov 03 22:08:59.309593 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309476 5615 cpu_manager.go:414] "containerMap: removing container" podUID="b037fb5a-8768-4e13-b190-b5226818fd81" containerName="maas-api" Nov 03 22:08:59.309593 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309480 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b037fb5a-8768-4e13-b190-b5226818fd81" containerName="maas-api" Nov 03 22:08:59.309593 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309489 5615 cpu_manager.go:414] "containerMap: removing container" podUID="ce51a38f-47a9-4926-adec-cecfceba6000" containerName="maas-api" Nov 03 22:08:59.309593 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309493 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce51a38f-47a9-4926-adec-cecfceba6000" containerName="maas-api" Nov 03 22:08:59.309593 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309504 5615 cpu_manager.go:414] "containerMap: removing container" podUID="5c90b432-31d5-49c0-8f4e-11397476f868" containerName="payload-processing" Nov 03 22:08:59.309593 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309509 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c90b432-31d5-49c0-8f4e-11397476f868" containerName="payload-processing" Nov 03 22:08:59.309593 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309516 5615 cpu_manager.go:414] "containerMap: removing container" podUID="a89b0627-f993-4003-ac8b-fc449b44d9d9" containerName="main" Nov 03 22:08:59.309593 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309523 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a89b0627-f993-4003-ac8b-fc449b44d9d9" containerName="main" Nov 03 22:08:59.309593 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309533 5615 cpu_manager.go:414] "containerMap: removing container" podUID="a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b" containerName="istio-proxy" Nov 03 22:08:59.309593 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309538 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b" containerName="istio-proxy" Nov 03 22:08:59.309593 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309552 5615 cpu_manager.go:414] "containerMap: removing container" podUID="0fd70b58-6f75-415c-b26e-303faf172e6c" containerName="istio-proxy" Nov 03 22:08:59.309593 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309560 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd70b58-6f75-415c-b26e-303faf172e6c" containerName="istio-proxy" Nov 03 22:08:59.309593 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309572 5615 cpu_manager.go:414] "containerMap: removing container" podUID="fd34f512-e1e0-433b-997f-b124daa6ded0" containerName="main" Nov 03 22:08:59.309593 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309579 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd34f512-e1e0-433b-997f-b124daa6ded0" containerName="main" Nov 03 22:08:59.309593 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309588 5615 cpu_manager.go:414] "containerMap: removing container" podUID="ef206c3a-762d-450d-9fb5-5a537392bffc" containerName="maas-api" Nov 03 22:08:59.309593 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309594 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef206c3a-762d-450d-9fb5-5a537392bffc" containerName="maas-api" Nov 03 22:08:59.309593 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309606 5615 cpu_manager.go:414] "containerMap: removing container" podUID="782d9d6a-583d-4bdc-af1e-d87d81db3d4a" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309613 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="782d9d6a-583d-4bdc-af1e-d87d81db3d4a" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309627 5615 cpu_manager.go:414] "containerMap: removing container" podUID="77818da7-0c48-434f-b2cb-68d47b3c2a33" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309634 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77818da7-0c48-434f-b2cb-68d47b3c2a33" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309644 5615 cpu_manager.go:414] "containerMap: removing container" podUID="9592ace4-6808-4b59-8bbc-58243dc78e12" containerName="maas-api" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309654 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9592ace4-6808-4b59-8bbc-58243dc78e12" containerName="maas-api" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309665 5615 cpu_manager.go:414] "containerMap: removing container" podUID="625d3189-ee5d-4264-b3a2-469ddecb4427" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309672 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="625d3189-ee5d-4264-b3a2-469ddecb4427" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309683 5615 cpu_manager.go:414] "containerMap: removing container" podUID="ad38eb17-97aa-4877-9efb-7e324f3ff5bd" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309690 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad38eb17-97aa-4877-9efb-7e324f3ff5bd" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309703 5615 cpu_manager.go:414] "containerMap: removing container" podUID="782d9d6a-583d-4bdc-af1e-d87d81db3d4a" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309710 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="782d9d6a-583d-4bdc-af1e-d87d81db3d4a" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309721 5615 cpu_manager.go:414] "containerMap: removing container" podUID="f3f0e38e-e1e7-451d-acc2-26df86d3c8ac" containerName="payload-pre-processing" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309728 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f0e38e-e1e7-451d-acc2-26df86d3c8ac" containerName="payload-pre-processing" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309740 5615 cpu_manager.go:414] "containerMap: removing container" podUID="ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309748 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309758 5615 cpu_manager.go:414] "containerMap: removing container" podUID="933ca863-b42a-4f84-9d67-b36634e0d9b4" containerName="maas-api" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309765 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="933ca863-b42a-4f84-9d67-b36634e0d9b4" containerName="maas-api" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309775 5615 cpu_manager.go:414] "containerMap: removing container" podUID="38461555-612e-478c-a9f8-98807b526d0f" containerName="payload-pre-processing" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309782 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38461555-612e-478c-a9f8-98807b526d0f" containerName="payload-pre-processing" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309792 5615 cpu_manager.go:414] "containerMap: removing container" podUID="232c2524-2557-47c5-9d80-f6235972ca83" containerName="main" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309798 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="232c2524-2557-47c5-9d80-f6235972ca83" containerName="main" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309811 5615 cpu_manager.go:414] "containerMap: removing container" podUID="98dbffe1-fe8a-43fa-821a-cfe49074a6a9" containerName="maas-api" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309818 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98dbffe1-fe8a-43fa-821a-cfe49074a6a9" containerName="maas-api" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309830 5615 cpu_manager.go:414] "containerMap: removing container" podUID="ad38eb17-97aa-4877-9efb-7e324f3ff5bd" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309837 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad38eb17-97aa-4877-9efb-7e324f3ff5bd" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309847 5615 cpu_manager.go:414] "containerMap: removing container" podUID="473b4351-c417-480c-9974-2f686f617433" containerName="main" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309854 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="473b4351-c417-480c-9974-2f686f617433" containerName="main" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309868 5615 cpu_manager.go:414] "containerMap: removing container" podUID="5611d11d-6bc4-4725-a5e7-d2cf9203f5ce" containerName="payload-processing" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309876 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5611d11d-6bc4-4725-a5e7-d2cf9203f5ce" containerName="payload-processing" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309887 5615 cpu_manager.go:414] "containerMap: removing container" podUID="dca66826-bea7-4c51-8fc0-b0e824ac5f2c" containerName="maas-api" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309893 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dca66826-bea7-4c51-8fc0-b0e824ac5f2c" containerName="maas-api" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309904 5615 cpu_manager.go:414] "containerMap: removing container" podUID="4271257f-77da-4bd1-9717-f7287bf6b713" containerName="maas-api" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309911 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4271257f-77da-4bd1-9717-f7287bf6b713" containerName="maas-api" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309921 5615 cpu_manager.go:414] "containerMap: removing container" podUID="ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309928 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309940 5615 cpu_manager.go:414] "containerMap: removing container" podUID="88203e05-6890-4c96-9d15-07c6685ad6d3" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309947 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88203e05-6890-4c96-9d15-07c6685ad6d3" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309959 5615 cpu_manager.go:414] "containerMap: removing container" podUID="39c75c51-eff9-4cb8-88b9-618dbb2ba188" containerName="main" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309966 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39c75c51-eff9-4cb8-88b9-618dbb2ba188" containerName="main" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309986 5615 cpu_manager.go:414] "containerMap: removing container" podUID="88203e05-6890-4c96-9d15-07c6685ad6d3" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.309993 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88203e05-6890-4c96-9d15-07c6685ad6d3" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310002 5615 cpu_manager.go:414] "containerMap: removing container" podUID="7803de3d-9e9c-475a-9227-74535c9c1a68" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310010 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7803de3d-9e9c-475a-9227-74535c9c1a68" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310129 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="77818da7-0c48-434f-b2cb-68d47b3c2a33" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310145 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="88203e05-6890-4c96-9d15-07c6685ad6d3" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310153 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="7803de3d-9e9c-475a-9227-74535c9c1a68" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310163 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd70b58-6f75-415c-b26e-303faf172e6c" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310173 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb894762-26e0-4dd6-9e55-0a0aed4455c9" containerName="payload-pre-processing" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310182 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="614c1da8-3cb5-4362-8774-b6b76ab79ac2" containerName="maas-api" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310191 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="b037fb5a-8768-4e13-b190-b5226818fd81" containerName="maas-api" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310198 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce51a38f-47a9-4926-adec-cecfceba6000" containerName="maas-api" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310208 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c90b432-31d5-49c0-8f4e-11397476f868" containerName="payload-processing" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310217 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="a89b0627-f993-4003-ac8b-fc449b44d9d9" containerName="main" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310227 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8d42f7c-c6eb-4ab3-a1d2-9d8a7572050b" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310237 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd34f512-e1e0-433b-997f-b124daa6ded0" containerName="main" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310246 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef206c3a-762d-450d-9fb5-5a537392bffc" containerName="maas-api" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310255 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="782d9d6a-583d-4bdc-af1e-d87d81db3d4a" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310263 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="77818da7-0c48-434f-b2cb-68d47b3c2a33" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310271 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="9592ace4-6808-4b59-8bbc-58243dc78e12" containerName="maas-api" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310280 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="625d3189-ee5d-4264-b3a2-469ddecb4427" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310289 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad38eb17-97aa-4877-9efb-7e324f3ff5bd" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310298 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3f0e38e-e1e7-451d-acc2-26df86d3c8ac" containerName="payload-pre-processing" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310305 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310313 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="933ca863-b42a-4f84-9d67-b36634e0d9b4" containerName="maas-api" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310322 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="38461555-612e-478c-a9f8-98807b526d0f" containerName="payload-pre-processing" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310332 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="232c2524-2557-47c5-9d80-f6235972ca83" containerName="main" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310340 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="98dbffe1-fe8a-43fa-821a-cfe49074a6a9" containerName="maas-api" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310348 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad38eb17-97aa-4877-9efb-7e324f3ff5bd" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310354 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="473b4351-c417-480c-9974-2f686f617433" containerName="main" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310363 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="5611d11d-6bc4-4725-a5e7-d2cf9203f5ce" containerName="payload-processing" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310375 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="dca66826-bea7-4c51-8fc0-b0e824ac5f2c" containerName="maas-api" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310383 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="4271257f-77da-4bd1-9717-f7287bf6b713" containerName="maas-api" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310390 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec2e19f3-34a2-4ab2-acdd-5da3cb8b1829" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310400 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="88203e05-6890-4c96-9d15-07c6685ad6d3" containerName="istio-proxy" Nov 03 22:08:59.310371 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.310409 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="39c75c51-eff9-4cb8-88b9-618dbb2ba188" containerName="main" Nov 03 22:08:59.315389 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.315370 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:08:59.353948 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.353911 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:08:59.354151 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.353956 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:08:59.354151 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.354064 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:08:59.354151 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.354120 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:08:59.354151 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.354144 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:08:59.354311 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.354244 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvbb2\" (UniqueName: \"kubernetes.io/projected/c6334c0a-3d27-416c-ab3c-ad1884cccce1-kube-api-access-gvbb2\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:08:59.354311 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.354279 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:08:59.354378 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.354320 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:08:59.354417 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.354372 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:08:59.369088 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.369052 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-099a5e-openshift-default-dockercfg-2h6nd\"" Nov 03 22:08:59.421579 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.421534 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2"] Nov 03 22:08:59.454940 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.454903 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:08:59.455137 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.454961 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:08:59.455137 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.454984 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:08:59.455137 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.455023 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:08:59.455137 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.455058 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:08:59.455137 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.455124 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:08:59.455484 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.455370 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:08:59.455484 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.455369 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:08:59.455484 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.455436 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:08:59.455484 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.455445 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:08:59.455720 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.455506 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvbb2\" (UniqueName: \"kubernetes.io/projected/c6334c0a-3d27-416c-ab3c-ad1884cccce1-kube-api-access-gvbb2\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:08:59.455720 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.455510 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:08:59.455720 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.455705 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:08:59.456102 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.456055 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:08:59.457359 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.457338 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:08:59.457517 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.457500 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:08:59.533632 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.533597 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:08:59.534119 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.534098 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvbb2\" (UniqueName: \"kubernetes.io/projected/c6334c0a-3d27-416c-ab3c-ad1884cccce1-kube-api-access-gvbb2\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:08:59.628528 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:08:59.628439 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:09:00.052153 ip-10-0-129-192 kubenswrapper[5615]: W1103 22:09:00.052121 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6334c0a_3d27_416c_ab3c_ad1884cccce1.slice/crio-2d880990e36ac5610646cfcf78f329b4093a7597b1b91dd761fb7dc0a54baae1 WatchSource:0}: Error finding container 2d880990e36ac5610646cfcf78f329b4093a7597b1b91dd761fb7dc0a54baae1: Status 404 returned error can't find the container with id 2d880990e36ac5610646cfcf78f329b4093a7597b1b91dd761fb7dc0a54baae1 Nov 03 22:09:00.053447 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:00.053422 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2"] Nov 03 22:09:00.054598 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:00.054566 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:09:00.054669 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:00.054631 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:09:00.054714 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:00.054679 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:09:00.457111 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:00.456999 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" event={"ID":"c6334c0a-3d27-416c-ab3c-ad1884cccce1","Type":"ContainerStarted","Data":"aecc85ef5afbf4c2ed95411e259dbb08667d5c9a20c83971a292ff0f9e4cb7b4"} Nov 03 22:09:00.457111 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:00.457066 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" event={"ID":"c6334c0a-3d27-416c-ab3c-ad1884cccce1","Type":"ContainerStarted","Data":"2d880990e36ac5610646cfcf78f329b4093a7597b1b91dd761fb7dc0a54baae1"} Nov 03 22:09:00.581103 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:00.581024 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" podStartSLOduration=1.5810086719999998 podStartE2EDuration="1.581008672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:08:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:09:00.58051717 +0000 UTC m=+4835.048295892" watchObservedRunningTime="2026-11-03 22:09:00.581008672 +0000 UTC m=+4835.048787397" Nov 03 22:09:00.629375 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:00.629329 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:09:01.630196 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:01.630158 5615 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 03 22:09:01.630610 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:01.630216 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" podUID="c6334c0a-3d27-416c-ab3c-ad1884cccce1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 03 22:09:02.586019 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:02.585970 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2"] Nov 03 22:09:02.629333 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:02.629291 5615 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 03 22:09:02.629528 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:02.629365 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" podUID="c6334c0a-3d27-416c-ab3c-ad1884cccce1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 03 22:09:03.471269 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:03.471202 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" podUID="c6334c0a-3d27-416c-ab3c-ad1884cccce1" containerName="istio-proxy" containerID="cri-o://aecc85ef5afbf4c2ed95411e259dbb08667d5c9a20c83971a292ff0f9e4cb7b4" gracePeriod=30 Nov 03 22:09:10.618335 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:09:10.618293 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:09:12.519097 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:12.519041 5615 generic.go:356] "Generic (PLEG): container finished" podID="c6334c0a-3d27-416c-ab3c-ad1884cccce1" containerID="aecc85ef5afbf4c2ed95411e259dbb08667d5c9a20c83971a292ff0f9e4cb7b4" exitCode=0 Nov 03 22:09:12.519488 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:12.519132 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" event={"ID":"c6334c0a-3d27-416c-ab3c-ad1884cccce1","Type":"ContainerDied","Data":"aecc85ef5afbf4c2ed95411e259dbb08667d5c9a20c83971a292ff0f9e4cb7b4"} Nov 03 22:09:13.023774 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.023746 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:09:13.115646 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.115611 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-workload-certs\") pod \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " Nov 03 22:09:13.115858 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.115708 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6334c0a-3d27-416c-ab3c-ad1884cccce1-kube-api-access-gvbb2\" (UniqueName: \"kubernetes.io/projected/c6334c0a-3d27-416c-ab3c-ad1884cccce1-kube-api-access-gvbb2\") pod \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " Nov 03 22:09:13.115858 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.115728 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-workload-socket\") pod \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " Nov 03 22:09:13.115858 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.115768 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istio-envoy\") pod \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " Nov 03 22:09:13.115858 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.115803 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istio-podinfo\") pod \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " Nov 03 22:09:13.115858 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.115847 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istio-token\" (UniqueName: \"kubernetes.io/projected/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istio-token\") pod \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " Nov 03 22:09:13.116177 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.115878 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istio-data\") pod \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " Nov 03 22:09:13.116177 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.115925 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istiod-ca-cert\") pod \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " Nov 03 22:09:13.116177 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.115945 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-workload-certs" pod "c6334c0a-3d27-416c-ab3c-ad1884cccce1" (UID: "c6334c0a-3d27-416c-ab3c-ad1884cccce1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:13.116177 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.115958 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-credential-socket\") pod \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\" (UID: \"c6334c0a-3d27-416c-ab3c-ad1884cccce1\") " Nov 03 22:09:13.116177 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.116053 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-workload-socket" pod "c6334c0a-3d27-416c-ab3c-ad1884cccce1" (UID: "c6334c0a-3d27-416c-ab3c-ad1884cccce1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:13.116177 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.116167 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-credential-socket" pod "c6334c0a-3d27-416c-ab3c-ad1884cccce1" (UID: "c6334c0a-3d27-416c-ab3c-ad1884cccce1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:13.116488 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.116371 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istio-data" pod "c6334c0a-3d27-416c-ab3c-ad1884cccce1" (UID: "c6334c0a-3d27-416c-ab3c-ad1884cccce1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:13.116544 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.116493 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:13.116544 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.116529 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:13.116544 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.116525 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istiod-ca-cert" pod "c6334c0a-3d27-416c-ab3c-ad1884cccce1" (UID: "c6334c0a-3d27-416c-ab3c-ad1884cccce1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:09:13.116696 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.116545 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:13.118389 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.118362 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istio-podinfo" pod "c6334c0a-3d27-416c-ab3c-ad1884cccce1" (UID: "c6334c0a-3d27-416c-ab3c-ad1884cccce1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:09:13.118589 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.118564 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istio-token" pod "c6334c0a-3d27-416c-ab3c-ad1884cccce1" (UID: "c6334c0a-3d27-416c-ab3c-ad1884cccce1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:09:13.118701 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.118683 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6334c0a-3d27-416c-ab3c-ad1884cccce1-kube-api-access-gvbb2" pod "c6334c0a-3d27-416c-ab3c-ad1884cccce1" (UID: "c6334c0a-3d27-416c-ab3c-ad1884cccce1"). InnerVolumeSpecName "kube-api-access-gvbb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:09:13.119545 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.119523 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istio-envoy" pod "c6334c0a-3d27-416c-ab3c-ad1884cccce1" (UID: "c6334c0a-3d27-416c-ab3c-ad1884cccce1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:13.217785 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.217702 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvbb2\" (UniqueName: \"kubernetes.io/projected/c6334c0a-3d27-416c-ab3c-ad1884cccce1-kube-api-access-gvbb2\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:13.217785 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.217737 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:13.217785 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.217747 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:13.217785 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.217755 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:13.217785 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.217765 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:13.217785 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.217773 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6334c0a-3d27-416c-ab3c-ad1884cccce1-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:13.527659 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.527630 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" Nov 03 22:09:13.527659 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.527635 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2" event={"ID":"c6334c0a-3d27-416c-ab3c-ad1884cccce1","Type":"ContainerDied","Data":"2d880990e36ac5610646cfcf78f329b4093a7597b1b91dd761fb7dc0a54baae1"} Nov 03 22:09:13.528209 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.527707 5615 scope.go:122] "RemoveContainer" containerID="aecc85ef5afbf4c2ed95411e259dbb08667d5c9a20c83971a292ff0f9e4cb7b4" Nov 03 22:09:13.695366 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.695327 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2"] Nov 03 22:09:13.699496 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:13.699464 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-wgcf2"] Nov 03 22:09:15.650039 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:15.650003 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6334c0a-3d27-416c-ab3c-ad1884cccce1" path="/var/lib/kubelet/pods/c6334c0a-3d27-416c-ab3c-ad1884cccce1/volumes" Nov 03 22:09:40.617986 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:09:40.617941 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:09:42.035609 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.035565 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx"] Nov 03 22:09:42.037851 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.036138 5615 cpu_manager.go:414] "containerMap: removing container" podUID="c6334c0a-3d27-416c-ab3c-ad1884cccce1" containerName="istio-proxy" Nov 03 22:09:42.037851 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.036160 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6334c0a-3d27-416c-ab3c-ad1884cccce1" containerName="istio-proxy" Nov 03 22:09:42.037851 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.036200 5615 cpu_manager.go:414] "containerMap: removing container" podUID="77818da7-0c48-434f-b2cb-68d47b3c2a33" containerName="istio-proxy" Nov 03 22:09:42.037851 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.036210 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77818da7-0c48-434f-b2cb-68d47b3c2a33" containerName="istio-proxy" Nov 03 22:09:42.037851 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.036312 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6334c0a-3d27-416c-ab3c-ad1884cccce1" containerName="istio-proxy" Nov 03 22:09:42.037851 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.036324 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd70b58-6f75-415c-b26e-303faf172e6c" containerName="istio-proxy" Nov 03 22:09:42.037851 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.036337 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="782d9d6a-583d-4bdc-af1e-d87d81db3d4a" containerName="istio-proxy" Nov 03 22:09:42.039334 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.039310 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:42.092930 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.092892 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-e84357-openshift-default-dockercfg-t4w57\"" Nov 03 22:09:42.101490 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.101462 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:42.101648 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.101501 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:42.101648 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.101520 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:42.101648 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.101553 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:42.101788 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.101680 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:42.101788 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.101706 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:42.101788 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.101744 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnp9h\" (UniqueName: \"kubernetes.io/projected/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-kube-api-access-jnp9h\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:42.101931 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.101847 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:42.101931 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.101865 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:42.145556 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.145516 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx"] Nov 03 22:09:42.203083 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.203036 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:42.203273 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.203088 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:42.203273 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.203128 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnp9h\" (UniqueName: \"kubernetes.io/projected/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-kube-api-access-jnp9h\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:42.203273 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.203189 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:42.203273 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.203213 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:42.203273 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.203260 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:42.203639 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.203291 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:42.203639 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.203315 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:42.203639 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.203351 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:42.203779 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.203643 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:42.203779 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.203677 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:42.203779 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.203725 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:42.203936 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.203789 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:42.204039 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.204021 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:42.205406 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.205378 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:42.205722 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.205703 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:42.265321 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.265284 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:42.265483 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.265347 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnp9h\" (UniqueName: \"kubernetes.io/projected/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-kube-api-access-jnp9h\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:42.353092 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.352990 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:42.770357 ip-10-0-129-192 kubenswrapper[5615]: W1103 22:09:42.770332 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4cb0b3e_cc37_42f2_8cd3_c07cf73b3139.slice/crio-1ad5312ad8af01443b8312d0232ca659f7fd425acc580998603244ab66aa46e2 WatchSource:0}: Error finding container 1ad5312ad8af01443b8312d0232ca659f7fd425acc580998603244ab66aa46e2: Status 404 returned error can't find the container with id 1ad5312ad8af01443b8312d0232ca659f7fd425acc580998603244ab66aa46e2 Nov 03 22:09:42.772059 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.772034 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx"] Nov 03 22:09:42.773007 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.772976 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:09:42.773050 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.773037 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:09:42.773113 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:42.773099 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:09:43.676786 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:43.676745 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" event={"ID":"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139","Type":"ContainerStarted","Data":"f8322cff31ca41f7325cf16f9b53671f5a7cdf6ca56278febe39acd48796e180"} Nov 03 22:09:43.677247 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:43.676810 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" event={"ID":"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139","Type":"ContainerStarted","Data":"1ad5312ad8af01443b8312d0232ca659f7fd425acc580998603244ab66aa46e2"} Nov 03 22:09:43.851350 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:43.851297 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" podStartSLOduration=2.851284353 podStartE2EDuration="2.851284353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:09:43.84971349 +0000 UTC m=+4878.317492225" watchObservedRunningTime="2026-11-03 22:09:43.851284353 +0000 UTC m=+4878.319063075" Nov 03 22:09:44.353816 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:44.353772 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:44.355505 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:44.355475 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Nov 03 22:09:44.355645 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:44.355528 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" podUID="f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Nov 03 22:09:45.354392 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:45.354352 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Nov 03 22:09:45.354814 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:45.354412 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" podUID="f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Nov 03 22:09:46.353409 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:46.353371 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Nov 03 22:09:46.353645 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:46.353437 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" podUID="f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Nov 03 22:09:47.353834 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:47.353794 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Nov 03 22:09:47.354280 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:47.353865 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" podUID="f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Nov 03 22:09:48.353758 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:48.353715 5615 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Nov 03 22:09:48.353932 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:48.353785 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" podUID="f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Nov 03 22:09:50.087586 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:50.087547 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:50.088069 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:50.087850 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:50.088796 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:50.088775 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:50.615988 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:50.615943 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx"] Nov 03 22:09:50.712027 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:50.711991 5615 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" secret="" err="secret \"e2e-worker-w4-e84357-openshift-default-dockercfg-t4w57\" not found" Nov 03 22:09:51.717136 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:51.717086 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" podUID="f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139" containerName="istio-proxy" containerID="cri-o://f8322cff31ca41f7325cf16f9b53671f5a7cdf6ca56278febe39acd48796e180" gracePeriod=30 Nov 03 22:09:55.613988 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:55.613954 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-b59f75cb9-vv6wb_20ef8f90-2fb3-4956-9aa6-cd9578f7c17b/manager/0.log" Nov 03 22:09:55.997729 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:55.997697 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-x87fj_79c0375b-54c7-4629-83da-f41b5b7e8da9/manager/0.log" Nov 03 22:09:56.235444 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:56.235413 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-9qt8v_42cba095-3e85-4cd0-a6fa-255c8086733b/manager/0.log" Nov 03 22:09:56.986566 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:56.986532 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:57.063976 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.063937 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-credential-socket\") pod \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " Nov 03 22:09:57.064184 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.064012 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istio-data\") pod \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " Nov 03 22:09:57.064184 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.064141 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istiod-ca-cert\") pod \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " Nov 03 22:09:57.064294 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.064184 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istio-token\" (UniqueName: \"kubernetes.io/projected/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istio-token\") pod \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " Nov 03 22:09:57.064294 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.064221 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istio-podinfo\") pod \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " Nov 03 22:09:57.064294 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.064256 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-workload-socket\") pod \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " Nov 03 22:09:57.064294 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.064282 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-workload-certs\") pod \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " Nov 03 22:09:57.064445 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.064312 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istio-envoy\") pod \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " Nov 03 22:09:57.064445 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.064363 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istio-data" pod "f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139" (UID: "f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:57.064445 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.064396 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-kube-api-access-jnp9h\" (UniqueName: \"kubernetes.io/projected/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-kube-api-access-jnp9h\") pod \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\" (UID: \"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139\") " Nov 03 22:09:57.064749 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.064730 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:57.064848 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.064831 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-credential-socket" pod "f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139" (UID: "f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:57.066501 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.066370 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-workload-certs" pod "f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139" (UID: "f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:57.066501 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.066381 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-workload-socket" pod "f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139" (UID: "f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:57.066501 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.066458 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istiod-ca-cert" pod "f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139" (UID: "f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:09:57.068386 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.068347 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istio-envoy" pod "f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139" (UID: "f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:57.068386 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.068380 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-kube-api-access-jnp9h" pod "f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139" (UID: "f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139"). InnerVolumeSpecName "kube-api-access-jnp9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:09:57.068626 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.068605 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istio-podinfo" pod "f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139" (UID: "f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:09:57.068671 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.068649 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istio-token" pod "f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139" (UID: "f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:09:57.165580 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.165537 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnp9h\" (UniqueName: \"kubernetes.io/projected/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-kube-api-access-jnp9h\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:57.165580 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.165578 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:57.165797 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.165590 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:57.165797 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.165599 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:57.165797 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.165608 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:57.165797 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.165616 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:57.165797 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.165625 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:57.165797 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.165633 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:57.229212 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.229180 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-lksbc_db528ef3-1b85-4e45-be0c-602a4eeb95c6/server/0.log" Nov 03 22:09:57.747087 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.747040 5615 generic.go:356] "Generic (PLEG): container finished" podID="f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139" containerID="f8322cff31ca41f7325cf16f9b53671f5a7cdf6ca56278febe39acd48796e180" exitCode=0 Nov 03 22:09:57.747313 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.747101 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" Nov 03 22:09:57.747313 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.747110 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" event={"ID":"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139","Type":"ContainerDied","Data":"f8322cff31ca41f7325cf16f9b53671f5a7cdf6ca56278febe39acd48796e180"} Nov 03 22:09:57.747313 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.747176 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx" event={"ID":"f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139","Type":"ContainerDied","Data":"1ad5312ad8af01443b8312d0232ca659f7fd425acc580998603244ab66aa46e2"} Nov 03 22:09:57.747313 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.747204 5615 scope.go:122] "RemoveContainer" containerID="f8322cff31ca41f7325cf16f9b53671f5a7cdf6ca56278febe39acd48796e180" Nov 03 22:09:57.756332 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.756313 5615 scope.go:122] "RemoveContainer" containerID="f8322cff31ca41f7325cf16f9b53671f5a7cdf6ca56278febe39acd48796e180" Nov 03 22:09:57.756622 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:09:57.756599 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8322cff31ca41f7325cf16f9b53671f5a7cdf6ca56278febe39acd48796e180\": container with ID starting with f8322cff31ca41f7325cf16f9b53671f5a7cdf6ca56278febe39acd48796e180 not found: ID does not exist" containerID="f8322cff31ca41f7325cf16f9b53671f5a7cdf6ca56278febe39acd48796e180" Nov 03 22:09:57.756668 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.756632 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8322cff31ca41f7325cf16f9b53671f5a7cdf6ca56278febe39acd48796e180"} err="failed to get container status \"f8322cff31ca41f7325cf16f9b53671f5a7cdf6ca56278febe39acd48796e180\": rpc error: code = NotFound desc = could not find container \"f8322cff31ca41f7325cf16f9b53671f5a7cdf6ca56278febe39acd48796e180\": container with ID starting with f8322cff31ca41f7325cf16f9b53671f5a7cdf6ca56278febe39acd48796e180 not found: ID does not exist" Nov 03 22:09:57.965833 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:57.965751 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx"] Nov 03 22:09:58.016464 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:58.016426 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-5w2bx"] Nov 03 22:09:59.650970 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:09:59.650935 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139" path="/var/lib/kubelet/pods/f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139/volumes" Nov 03 22:10:10.618794 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:10:10.618748 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:10:12.030600 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:12.030563 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r"] Nov 03 22:10:12.031115 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:12.031006 5615 cpu_manager.go:414] "containerMap: removing container" podUID="f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139" containerName="istio-proxy" Nov 03 22:10:12.031115 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:12.031022 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139" containerName="istio-proxy" Nov 03 22:10:12.031242 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:12.031133 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4cb0b3e-cc37-42f2-8cd3-c07cf73b3139" containerName="istio-proxy" Nov 03 22:10:12.038024 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:12.038001 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r" Nov 03 22:10:12.093515 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:12.093480 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r"] Nov 03 22:10:12.110558 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:12.110511 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8a32080-ef43-4862-92d4-7a5db9f10519-maas-api-tls\") pod \"maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r\" (UID: \"f8a32080-ef43-4862-92d4-7a5db9f10519\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r" Nov 03 22:10:12.110741 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:12.110624 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a32080-ef43-4862-92d4-7a5db9f10519-openshift-service-ca\") pod \"maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r\" (UID: \"f8a32080-ef43-4862-92d4-7a5db9f10519\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r" Nov 03 22:10:12.110831 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:12.110809 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9phg9\" (UniqueName: \"kubernetes.io/projected/f8a32080-ef43-4862-92d4-7a5db9f10519-kube-api-access-9phg9\") pod \"maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r\" (UID: \"f8a32080-ef43-4862-92d4-7a5db9f10519\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r" Nov 03 22:10:12.110883 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:12.110853 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8a32080-ef43-4862-92d4-7a5db9f10519-metrics-certs\") pod \"maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r\" (UID: \"f8a32080-ef43-4862-92d4-7a5db9f10519\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r" Nov 03 22:10:12.142521 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:12.142478 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4edfb8c0-serving-cert\"" Nov 03 22:10:12.211392 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:12.211348 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9phg9\" (UniqueName: \"kubernetes.io/projected/f8a32080-ef43-4862-92d4-7a5db9f10519-kube-api-access-9phg9\") pod \"maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r\" (UID: \"f8a32080-ef43-4862-92d4-7a5db9f10519\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r" Nov 03 22:10:12.211392 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:12.211390 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8a32080-ef43-4862-92d4-7a5db9f10519-metrics-certs\") pod \"maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r\" (UID: \"f8a32080-ef43-4862-92d4-7a5db9f10519\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r" Nov 03 22:10:12.211637 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:12.211430 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8a32080-ef43-4862-92d4-7a5db9f10519-maas-api-tls\") pod \"maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r\" (UID: \"f8a32080-ef43-4862-92d4-7a5db9f10519\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r" Nov 03 22:10:12.211637 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:12.211451 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a32080-ef43-4862-92d4-7a5db9f10519-openshift-service-ca\") pod \"maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r\" (UID: \"f8a32080-ef43-4862-92d4-7a5db9f10519\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r" Nov 03 22:10:12.211637 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:10:12.211572 5615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-serving-cert: secret "maas-api-e2e-ait-4edfb8c0-serving-cert" not found Nov 03 22:10:12.211762 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:10:12.211637 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8a32080-ef43-4862-92d4-7a5db9f10519-maas-api-tls podName:f8a32080-ef43-4862-92d4-7a5db9f10519 nodeName:}" failed. No retries permitted until 2026-11-03 22:10:12.711619866 +0000 UTC m=+4907.179398588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f8a32080-ef43-4862-92d4-7a5db9f10519-maas-api-tls") pod "maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r" (UID: "f8a32080-ef43-4862-92d4-7a5db9f10519") : secret "maas-api-e2e-ait-4edfb8c0-serving-cert" not found Nov 03 22:10:12.212237 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:12.212215 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a32080-ef43-4862-92d4-7a5db9f10519-openshift-service-ca\") pod \"maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r\" (UID: \"f8a32080-ef43-4862-92d4-7a5db9f10519\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r" Nov 03 22:10:12.213966 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:12.213943 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8a32080-ef43-4862-92d4-7a5db9f10519-metrics-certs\") pod \"maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r\" (UID: \"f8a32080-ef43-4862-92d4-7a5db9f10519\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r" Nov 03 22:10:12.274119 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:12.274057 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9phg9\" (UniqueName: \"kubernetes.io/projected/f8a32080-ef43-4862-92d4-7a5db9f10519-kube-api-access-9phg9\") pod \"maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r\" (UID: \"f8a32080-ef43-4862-92d4-7a5db9f10519\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r" Nov 03 22:10:12.715435 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:12.715388 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8a32080-ef43-4862-92d4-7a5db9f10519-maas-api-tls\") pod \"maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r\" (UID: \"f8a32080-ef43-4862-92d4-7a5db9f10519\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r" Nov 03 22:10:12.718023 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:12.717991 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8a32080-ef43-4862-92d4-7a5db9f10519-maas-api-tls\") pod \"maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r\" (UID: \"f8a32080-ef43-4862-92d4-7a5db9f10519\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r" Nov 03 22:10:12.951289 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:12.951243 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r" Nov 03 22:10:13.326653 ip-10-0-129-192 kubenswrapper[5615]: W1103 22:10:13.326615 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a32080_ef43_4862_92d4_7a5db9f10519.slice/crio-ce85f0dc0b304f93eb78b990ab62ee71d885cf1104a5b6eeaaba254050724fb5 WatchSource:0}: Error finding container ce85f0dc0b304f93eb78b990ab62ee71d885cf1104a5b6eeaaba254050724fb5: Status 404 returned error can't find the container with id ce85f0dc0b304f93eb78b990ab62ee71d885cf1104a5b6eeaaba254050724fb5 Nov 03 22:10:13.328099 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:13.328052 5615 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:10:13.328355 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:13.328333 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r"] Nov 03 22:10:13.824143 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:13.824102 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r" event={"ID":"f8a32080-ef43-4862-92d4-7a5db9f10519","Type":"ContainerStarted","Data":"ce85f0dc0b304f93eb78b990ab62ee71d885cf1104a5b6eeaaba254050724fb5"} Nov 03 22:10:14.834259 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:14.834216 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r" event={"ID":"f8a32080-ef43-4862-92d4-7a5db9f10519","Type":"ContainerStarted","Data":"4e7ebe7aedd21472b851c6cbfe450f27f0834c8d13367cfb81ae8d522ed2088d"} Nov 03 22:10:14.834668 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:14.834296 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r" Nov 03 22:10:14.954278 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:14.954226 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r" podStartSLOduration=3.029937125 podStartE2EDuration="3.954211893s" totalImagesPullingTime="924.274768ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:10:14.953779978 +0000 UTC m=+4909.421558701" watchObservedRunningTime="2026-11-03 22:10:14.954211893 +0000 UTC m=+4909.421990616" Nov 03 22:10:20.845115 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:20.845064 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r" Nov 03 22:10:40.618587 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:10:40.618543 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:10:57.885635 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:57.885598 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r"] Nov 03 22:10:57.886119 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:57.885819 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r" podUID="f8a32080-ef43-4862-92d4-7a5db9f10519" containerName="maas-api" containerID="cri-o://4e7ebe7aedd21472b851c6cbfe450f27f0834c8d13367cfb81ae8d522ed2088d" gracePeriod=30 Nov 03 22:10:58.047365 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:58.047330 5615 generic.go:356] "Generic (PLEG): container finished" podID="f8a32080-ef43-4862-92d4-7a5db9f10519" containerID="4e7ebe7aedd21472b851c6cbfe450f27f0834c8d13367cfb81ae8d522ed2088d" exitCode=0 Nov 03 22:10:58.047528 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:58.047399 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r" event={"ID":"f8a32080-ef43-4862-92d4-7a5db9f10519","Type":"ContainerDied","Data":"4e7ebe7aedd21472b851c6cbfe450f27f0834c8d13367cfb81ae8d522ed2088d"} Nov 03 22:10:58.141629 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:58.141568 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r" Nov 03 22:10:58.173739 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:58.173707 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8a32080-ef43-4862-92d4-7a5db9f10519-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a32080-ef43-4862-92d4-7a5db9f10519-openshift-service-ca\") pod \"f8a32080-ef43-4862-92d4-7a5db9f10519\" (UID: \"f8a32080-ef43-4862-92d4-7a5db9f10519\") " Nov 03 22:10:58.173922 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:58.173811 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8a32080-ef43-4862-92d4-7a5db9f10519-kube-api-access-9phg9\" (UniqueName: \"kubernetes.io/projected/f8a32080-ef43-4862-92d4-7a5db9f10519-kube-api-access-9phg9\") pod \"f8a32080-ef43-4862-92d4-7a5db9f10519\" (UID: \"f8a32080-ef43-4862-92d4-7a5db9f10519\") " Nov 03 22:10:58.173922 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:58.173899 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8a32080-ef43-4862-92d4-7a5db9f10519-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8a32080-ef43-4862-92d4-7a5db9f10519-metrics-certs\") pod \"f8a32080-ef43-4862-92d4-7a5db9f10519\" (UID: \"f8a32080-ef43-4862-92d4-7a5db9f10519\") " Nov 03 22:10:58.174007 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:58.173951 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8a32080-ef43-4862-92d4-7a5db9f10519-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8a32080-ef43-4862-92d4-7a5db9f10519-maas-api-tls\") pod \"f8a32080-ef43-4862-92d4-7a5db9f10519\" (UID: \"f8a32080-ef43-4862-92d4-7a5db9f10519\") " Nov 03 22:10:58.174176 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:58.174154 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8a32080-ef43-4862-92d4-7a5db9f10519-openshift-service-ca" pod "f8a32080-ef43-4862-92d4-7a5db9f10519" (UID: "f8a32080-ef43-4862-92d4-7a5db9f10519"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:10:58.174333 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:58.174320 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a32080-ef43-4862-92d4-7a5db9f10519-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:10:58.176024 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:58.175993 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8a32080-ef43-4862-92d4-7a5db9f10519-maas-api-tls" pod "f8a32080-ef43-4862-92d4-7a5db9f10519" (UID: "f8a32080-ef43-4862-92d4-7a5db9f10519"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:10:58.176024 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:58.176013 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8a32080-ef43-4862-92d4-7a5db9f10519-metrics-certs" pod "f8a32080-ef43-4862-92d4-7a5db9f10519" (UID: "f8a32080-ef43-4862-92d4-7a5db9f10519"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:10:58.176192 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:58.175993 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a32080-ef43-4862-92d4-7a5db9f10519-kube-api-access-9phg9" pod "f8a32080-ef43-4862-92d4-7a5db9f10519" (UID: "f8a32080-ef43-4862-92d4-7a5db9f10519"). InnerVolumeSpecName "kube-api-access-9phg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:10:58.275724 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:58.275685 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9phg9\" (UniqueName: \"kubernetes.io/projected/f8a32080-ef43-4862-92d4-7a5db9f10519-kube-api-access-9phg9\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:10:58.275724 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:58.275717 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8a32080-ef43-4862-92d4-7a5db9f10519-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:10:58.275724 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:58.275727 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8a32080-ef43-4862-92d4-7a5db9f10519-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:10:59.053168 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:59.053135 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r" Nov 03 22:10:59.053653 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:59.053133 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r" event={"ID":"f8a32080-ef43-4862-92d4-7a5db9f10519","Type":"ContainerDied","Data":"ce85f0dc0b304f93eb78b990ab62ee71d885cf1104a5b6eeaaba254050724fb5"} Nov 03 22:10:59.053653 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:59.053269 5615 scope.go:122] "RemoveContainer" containerID="4e7ebe7aedd21472b851c6cbfe450f27f0834c8d13367cfb81ae8d522ed2088d" Nov 03 22:10:59.219986 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:59.219954 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r"] Nov 03 22:10:59.271200 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:59.271166 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4edfb8c0-d9487dd87-hsc4r"] Nov 03 22:10:59.650515 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:10:59.650478 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8a32080-ef43-4862-92d4-7a5db9f10519" path="/var/lib/kubelet/pods/f8a32080-ef43-4862-92d4-7a5db9f10519/volumes" Nov 03 22:11:10.626624 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:11:10.626531 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:11:27.922779 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:27.922742 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn"] Nov 03 22:11:27.923456 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:27.923163 5615 cpu_manager.go:414] "containerMap: removing container" podUID="f8a32080-ef43-4862-92d4-7a5db9f10519" containerName="maas-api" Nov 03 22:11:27.923456 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:27.923174 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8a32080-ef43-4862-92d4-7a5db9f10519" containerName="maas-api" Nov 03 22:11:27.923456 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:27.923264 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8a32080-ef43-4862-92d4-7a5db9f10519" containerName="maas-api" Nov 03 22:11:27.927615 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:27.927595 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:27.964829 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:27.964741 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b73a9a0-b746-4c20-bfae-f3c670ef04cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q9mnn\" (UID: \"6b73a9a0-b746-4c20-bfae-f3c670ef04cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:27.964829 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:27.964796 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b73a9a0-b746-4c20-bfae-f3c670ef04cf-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q9mnn\" (UID: \"6b73a9a0-b746-4c20-bfae-f3c670ef04cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:27.965088 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:27.964935 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b73a9a0-b746-4c20-bfae-f3c670ef04cf-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q9mnn\" (UID: \"6b73a9a0-b746-4c20-bfae-f3c670ef04cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:27.965088 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:27.964995 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b73a9a0-b746-4c20-bfae-f3c670ef04cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q9mnn\" (UID: \"6b73a9a0-b746-4c20-bfae-f3c670ef04cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:27.965217 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:27.965107 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b73a9a0-b746-4c20-bfae-f3c670ef04cf-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q9mnn\" (UID: \"6b73a9a0-b746-4c20-bfae-f3c670ef04cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:27.965217 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:27.965165 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b73a9a0-b746-4c20-bfae-f3c670ef04cf-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q9mnn\" (UID: \"6b73a9a0-b746-4c20-bfae-f3c670ef04cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:27.965305 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:27.965238 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28fwg\" (UniqueName: \"kubernetes.io/projected/6b73a9a0-b746-4c20-bfae-f3c670ef04cf-kube-api-access-28fwg\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q9mnn\" (UID: \"6b73a9a0-b746-4c20-bfae-f3c670ef04cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:27.965305 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:27.965291 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b73a9a0-b746-4c20-bfae-f3c670ef04cf-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q9mnn\" (UID: \"6b73a9a0-b746-4c20-bfae-f3c670ef04cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:27.966247 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:27.965377 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b73a9a0-b746-4c20-bfae-f3c670ef04cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q9mnn\" (UID: \"6b73a9a0-b746-4c20-bfae-f3c670ef04cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:28.009400 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:28.009357 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn"] Nov 03 22:11:28.066225 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:28.066191 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b73a9a0-b746-4c20-bfae-f3c670ef04cf-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q9mnn\" (UID: \"6b73a9a0-b746-4c20-bfae-f3c670ef04cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:28.066437 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:28.066238 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b73a9a0-b746-4c20-bfae-f3c670ef04cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q9mnn\" (UID: \"6b73a9a0-b746-4c20-bfae-f3c670ef04cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:28.066437 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:28.066267 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b73a9a0-b746-4c20-bfae-f3c670ef04cf-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q9mnn\" (UID: \"6b73a9a0-b746-4c20-bfae-f3c670ef04cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:28.066437 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:28.066306 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b73a9a0-b746-4c20-bfae-f3c670ef04cf-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q9mnn\" (UID: \"6b73a9a0-b746-4c20-bfae-f3c670ef04cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:28.066437 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:28.066332 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28fwg\" (UniqueName: \"kubernetes.io/projected/6b73a9a0-b746-4c20-bfae-f3c670ef04cf-kube-api-access-28fwg\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q9mnn\" (UID: \"6b73a9a0-b746-4c20-bfae-f3c670ef04cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:28.066437 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:28.066370 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b73a9a0-b746-4c20-bfae-f3c670ef04cf-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q9mnn\" (UID: \"6b73a9a0-b746-4c20-bfae-f3c670ef04cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:28.066437 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:28.066398 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b73a9a0-b746-4c20-bfae-f3c670ef04cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q9mnn\" (UID: \"6b73a9a0-b746-4c20-bfae-f3c670ef04cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:28.066778 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:28.066457 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b73a9a0-b746-4c20-bfae-f3c670ef04cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q9mnn\" (UID: \"6b73a9a0-b746-4c20-bfae-f3c670ef04cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:28.066778 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:28.066500 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b73a9a0-b746-4c20-bfae-f3c670ef04cf-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q9mnn\" (UID: \"6b73a9a0-b746-4c20-bfae-f3c670ef04cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:28.066778 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:28.066667 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b73a9a0-b746-4c20-bfae-f3c670ef04cf-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q9mnn\" (UID: \"6b73a9a0-b746-4c20-bfae-f3c670ef04cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:28.066778 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:28.066767 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b73a9a0-b746-4c20-bfae-f3c670ef04cf-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q9mnn\" (UID: \"6b73a9a0-b746-4c20-bfae-f3c670ef04cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:28.066999 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:28.066816 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b73a9a0-b746-4c20-bfae-f3c670ef04cf-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q9mnn\" (UID: \"6b73a9a0-b746-4c20-bfae-f3c670ef04cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:28.066999 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:28.066948 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b73a9a0-b746-4c20-bfae-f3c670ef04cf-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q9mnn\" (UID: \"6b73a9a0-b746-4c20-bfae-f3c670ef04cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:28.067411 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:28.067385 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b73a9a0-b746-4c20-bfae-f3c670ef04cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q9mnn\" (UID: \"6b73a9a0-b746-4c20-bfae-f3c670ef04cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:28.068775 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:28.068751 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b73a9a0-b746-4c20-bfae-f3c670ef04cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q9mnn\" (UID: \"6b73a9a0-b746-4c20-bfae-f3c670ef04cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:28.069109 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:28.069062 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b73a9a0-b746-4c20-bfae-f3c670ef04cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q9mnn\" (UID: \"6b73a9a0-b746-4c20-bfae-f3c670ef04cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:28.128656 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:28.128621 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b73a9a0-b746-4c20-bfae-f3c670ef04cf-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q9mnn\" (UID: \"6b73a9a0-b746-4c20-bfae-f3c670ef04cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:28.128847 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:28.128763 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28fwg\" (UniqueName: \"kubernetes.io/projected/6b73a9a0-b746-4c20-bfae-f3c670ef04cf-kube-api-access-28fwg\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q9mnn\" (UID: \"6b73a9a0-b746-4c20-bfae-f3c670ef04cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:28.240949 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:28.240904 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:28.660929 ip-10-0-129-192 kubenswrapper[5615]: W1103 22:11:28.660900 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b73a9a0_b746_4c20_bfae_f3c670ef04cf.slice/crio-4b9ebd5bf570cfd102672e056a60fdba95bbe12c2d728acaecd02177c2ad97b9 WatchSource:0}: Error finding container 4b9ebd5bf570cfd102672e056a60fdba95bbe12c2d728acaecd02177c2ad97b9: Status 404 returned error can't find the container with id 4b9ebd5bf570cfd102672e056a60fdba95bbe12c2d728acaecd02177c2ad97b9 Nov 03 22:11:28.662092 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:28.662052 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn"] Nov 03 22:11:28.663150 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:28.663115 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:11:28.663234 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:28.663181 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:11:28.663234 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:28.663210 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:11:29.196518 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:29.196476 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" event={"ID":"6b73a9a0-b746-4c20-bfae-f3c670ef04cf","Type":"ContainerStarted","Data":"0a96e9aeb041236956b496b9ae5c47e267a997d78054118cc7f6a2af4048197c"} Nov 03 22:11:29.196985 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:29.196531 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" event={"ID":"6b73a9a0-b746-4c20-bfae-f3c670ef04cf","Type":"ContainerStarted","Data":"4b9ebd5bf570cfd102672e056a60fdba95bbe12c2d728acaecd02177c2ad97b9"} Nov 03 22:11:29.241515 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:29.241472 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:29.243150 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:29.243123 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Nov 03 22:11:29.243339 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:29.243181 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" podUID="6b73a9a0-b746-4c20-bfae-f3c670ef04cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Nov 03 22:11:29.323069 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:29.323013 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" podStartSLOduration=2.322997228 podStartE2EDuration="2.322997228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:11:29.322550613 +0000 UTC m=+4983.790329403" watchObservedRunningTime="2026-11-03 22:11:29.322997228 +0000 UTC m=+4983.790776011" Nov 03 22:11:30.241422 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:30.241381 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Nov 03 22:11:30.241879 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:30.241443 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" podUID="6b73a9a0-b746-4c20-bfae-f3c670ef04cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Nov 03 22:11:31.242172 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:31.242130 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Nov 03 22:11:31.242617 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:31.242188 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" podUID="6b73a9a0-b746-4c20-bfae-f3c670ef04cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Nov 03 22:11:32.241745 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:32.241709 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Nov 03 22:11:32.241927 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:32.241774 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" podUID="6b73a9a0-b746-4c20-bfae-f3c670ef04cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Nov 03 22:11:33.241457 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:33.241422 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Nov 03 22:11:33.241990 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:33.241489 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" podUID="6b73a9a0-b746-4c20-bfae-f3c670ef04cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Nov 03 22:11:34.241894 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:34.241857 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Nov 03 22:11:34.242468 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:34.241918 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" podUID="6b73a9a0-b746-4c20-bfae-f3c670ef04cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Nov 03 22:11:35.241523 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:35.241473 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Nov 03 22:11:35.241717 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:35.241557 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" podUID="6b73a9a0-b746-4c20-bfae-f3c670ef04cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Nov 03 22:11:36.241705 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:36.241662 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Nov 03 22:11:36.242166 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:36.241732 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" podUID="6b73a9a0-b746-4c20-bfae-f3c670ef04cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Nov 03 22:11:37.245043 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:37.245007 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:37.245495 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:37.245343 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:37.245968 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:11:37.245949 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" Nov 03 22:11:40.624868 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:11:40.624819 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:12:10.618224 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:12:10.618179 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:12:28.398674 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.398635 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp"] Nov 03 22:12:28.402278 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.402261 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:28.497123 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.497062 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0aabe10-5135-4ca5-91a2-226a31408d6b-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5q7qp\" (UID: \"e0aabe10-5135-4ca5-91a2-226a31408d6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:28.497326 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.497144 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0aabe10-5135-4ca5-91a2-226a31408d6b-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5q7qp\" (UID: \"e0aabe10-5135-4ca5-91a2-226a31408d6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:28.497405 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.497311 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0aabe10-5135-4ca5-91a2-226a31408d6b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5q7qp\" (UID: \"e0aabe10-5135-4ca5-91a2-226a31408d6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:28.497467 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.497422 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0aabe10-5135-4ca5-91a2-226a31408d6b-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5q7qp\" (UID: \"e0aabe10-5135-4ca5-91a2-226a31408d6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:28.497467 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.497452 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0aabe10-5135-4ca5-91a2-226a31408d6b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5q7qp\" (UID: \"e0aabe10-5135-4ca5-91a2-226a31408d6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:28.497592 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.497481 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0aabe10-5135-4ca5-91a2-226a31408d6b-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5q7qp\" (UID: \"e0aabe10-5135-4ca5-91a2-226a31408d6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:28.497592 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.497531 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0aabe10-5135-4ca5-91a2-226a31408d6b-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5q7qp\" (UID: \"e0aabe10-5135-4ca5-91a2-226a31408d6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:28.497592 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.497575 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0aabe10-5135-4ca5-91a2-226a31408d6b-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5q7qp\" (UID: \"e0aabe10-5135-4ca5-91a2-226a31408d6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:28.497758 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.497648 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v552x\" (UniqueName: \"kubernetes.io/projected/e0aabe10-5135-4ca5-91a2-226a31408d6b-kube-api-access-v552x\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5q7qp\" (UID: \"e0aabe10-5135-4ca5-91a2-226a31408d6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:28.508711 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.508679 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp"] Nov 03 22:12:28.598343 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.598306 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v552x\" (UniqueName: \"kubernetes.io/projected/e0aabe10-5135-4ca5-91a2-226a31408d6b-kube-api-access-v552x\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5q7qp\" (UID: \"e0aabe10-5135-4ca5-91a2-226a31408d6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:28.598561 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.598366 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0aabe10-5135-4ca5-91a2-226a31408d6b-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5q7qp\" (UID: \"e0aabe10-5135-4ca5-91a2-226a31408d6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:28.598561 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.598397 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0aabe10-5135-4ca5-91a2-226a31408d6b-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5q7qp\" (UID: \"e0aabe10-5135-4ca5-91a2-226a31408d6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:28.598561 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.598423 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0aabe10-5135-4ca5-91a2-226a31408d6b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5q7qp\" (UID: \"e0aabe10-5135-4ca5-91a2-226a31408d6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:28.598561 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.598500 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0aabe10-5135-4ca5-91a2-226a31408d6b-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5q7qp\" (UID: \"e0aabe10-5135-4ca5-91a2-226a31408d6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:28.598561 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.598536 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0aabe10-5135-4ca5-91a2-226a31408d6b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5q7qp\" (UID: \"e0aabe10-5135-4ca5-91a2-226a31408d6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:28.598561 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.598555 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0aabe10-5135-4ca5-91a2-226a31408d6b-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5q7qp\" (UID: \"e0aabe10-5135-4ca5-91a2-226a31408d6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:28.598866 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.598577 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0aabe10-5135-4ca5-91a2-226a31408d6b-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5q7qp\" (UID: \"e0aabe10-5135-4ca5-91a2-226a31408d6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:28.598866 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.598616 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0aabe10-5135-4ca5-91a2-226a31408d6b-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5q7qp\" (UID: \"e0aabe10-5135-4ca5-91a2-226a31408d6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:28.598983 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.598876 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0aabe10-5135-4ca5-91a2-226a31408d6b-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5q7qp\" (UID: \"e0aabe10-5135-4ca5-91a2-226a31408d6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:28.598983 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.598928 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0aabe10-5135-4ca5-91a2-226a31408d6b-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5q7qp\" (UID: \"e0aabe10-5135-4ca5-91a2-226a31408d6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:28.599134 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.599029 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0aabe10-5135-4ca5-91a2-226a31408d6b-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5q7qp\" (UID: \"e0aabe10-5135-4ca5-91a2-226a31408d6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:28.599205 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.599182 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0aabe10-5135-4ca5-91a2-226a31408d6b-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5q7qp\" (UID: \"e0aabe10-5135-4ca5-91a2-226a31408d6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:28.599352 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.599331 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0aabe10-5135-4ca5-91a2-226a31408d6b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5q7qp\" (UID: \"e0aabe10-5135-4ca5-91a2-226a31408d6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:28.600846 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.600821 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0aabe10-5135-4ca5-91a2-226a31408d6b-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5q7qp\" (UID: \"e0aabe10-5135-4ca5-91a2-226a31408d6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:28.601108 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.601068 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0aabe10-5135-4ca5-91a2-226a31408d6b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5q7qp\" (UID: \"e0aabe10-5135-4ca5-91a2-226a31408d6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:28.660750 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.660677 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0aabe10-5135-4ca5-91a2-226a31408d6b-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5q7qp\" (UID: \"e0aabe10-5135-4ca5-91a2-226a31408d6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:28.660918 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.660782 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v552x\" (UniqueName: \"kubernetes.io/projected/e0aabe10-5135-4ca5-91a2-226a31408d6b-kube-api-access-v552x\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5q7qp\" (UID: \"e0aabe10-5135-4ca5-91a2-226a31408d6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:28.716536 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:28.716497 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:29.137840 ip-10-0-129-192 kubenswrapper[5615]: W1103 22:12:29.137794 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0aabe10_5135_4ca5_91a2_226a31408d6b.slice/crio-33a3a4f0c968b24edaa2320b2e3c0dc4b5fc7126da378ea01637c1f894fba511 WatchSource:0}: Error finding container 33a3a4f0c968b24edaa2320b2e3c0dc4b5fc7126da378ea01637c1f894fba511: Status 404 returned error can't find the container with id 33a3a4f0c968b24edaa2320b2e3c0dc4b5fc7126da378ea01637c1f894fba511 Nov 03 22:12:29.138907 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:29.138879 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp"] Nov 03 22:12:29.140330 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:29.140295 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:12:29.140412 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:29.140361 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:12:29.140412 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:29.140391 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:12:29.508509 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:29.508471 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" event={"ID":"e0aabe10-5135-4ca5-91a2-226a31408d6b","Type":"ContainerStarted","Data":"53a5d5a1bb423ad13053381af9775334c89f2e90d86e20d8fca5554f99747b6d"} Nov 03 22:12:29.508976 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:29.508531 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" event={"ID":"e0aabe10-5135-4ca5-91a2-226a31408d6b","Type":"ContainerStarted","Data":"33a3a4f0c968b24edaa2320b2e3c0dc4b5fc7126da378ea01637c1f894fba511"} Nov 03 22:12:29.632344 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:29.632282 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" podStartSLOduration=1.632264411 podStartE2EDuration="1.632264411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:29.631261277 +0000 UTC m=+5044.099040000" watchObservedRunningTime="2026-11-03 22:12:29.632264411 +0000 UTC m=+5044.100043133" Nov 03 22:12:29.719097 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:29.716877 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:30.722102 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:30.721105 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 03 22:12:30.722102 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:30.721213 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" podUID="e0aabe10-5135-4ca5-91a2-226a31408d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 03 22:12:31.719100 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:31.718570 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 03 22:12:31.719100 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:31.718652 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" podUID="e0aabe10-5135-4ca5-91a2-226a31408d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 03 22:12:32.722100 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:32.721573 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 03 22:12:32.722100 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:32.721657 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" podUID="e0aabe10-5135-4ca5-91a2-226a31408d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 03 22:12:33.719099 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:33.718475 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 03 22:12:33.719099 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:33.718554 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" podUID="e0aabe10-5135-4ca5-91a2-226a31408d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 03 22:12:34.720099 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:34.719534 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 03 22:12:34.720099 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:34.719619 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" podUID="e0aabe10-5135-4ca5-91a2-226a31408d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 03 22:12:35.717496 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:35.717410 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 03 22:12:35.717662 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:35.717485 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" podUID="e0aabe10-5135-4ca5-91a2-226a31408d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 03 22:12:37.718626 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:37.718573 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:12:37.719042 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:37.718657 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" podUID="e0aabe10-5135-4ca5-91a2-226a31408d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:12:38.760323 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:38.760273 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:12:38.760794 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:38.760354 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" podUID="e0aabe10-5135-4ca5-91a2-226a31408d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:12:39.624455 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:39.624422 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:39.624838 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:39.624815 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:39.625561 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:39.625532 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" Nov 03 22:12:40.626180 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:12:40.626137 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:12:58.644836 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.644792 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f"] Nov 03 22:12:58.649089 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.649051 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:12:58.715577 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.715543 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f"] Nov 03 22:12:58.756106 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.756037 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f7ca28-e370-4e4c-85f3-818e70156336-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-r666f\" (UID: \"87f7ca28-e370-4e4c-85f3-818e70156336\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:12:58.756298 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.756116 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87f7ca28-e370-4e4c-85f3-818e70156336-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-r666f\" (UID: \"87f7ca28-e370-4e4c-85f3-818e70156336\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:12:58.756298 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.756253 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f7ca28-e370-4e4c-85f3-818e70156336-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-r666f\" (UID: \"87f7ca28-e370-4e4c-85f3-818e70156336\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:12:58.756298 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.756286 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87f7ca28-e370-4e4c-85f3-818e70156336-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-r666f\" (UID: \"87f7ca28-e370-4e4c-85f3-818e70156336\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:12:58.756479 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.756317 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87f7ca28-e370-4e4c-85f3-818e70156336-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-r666f\" (UID: \"87f7ca28-e370-4e4c-85f3-818e70156336\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:12:58.756479 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.756354 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87f7ca28-e370-4e4c-85f3-818e70156336-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-r666f\" (UID: \"87f7ca28-e370-4e4c-85f3-818e70156336\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:12:58.756479 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.756412 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87f7ca28-e370-4e4c-85f3-818e70156336-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-r666f\" (UID: \"87f7ca28-e370-4e4c-85f3-818e70156336\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:12:58.756479 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.756451 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87f7ca28-e370-4e4c-85f3-818e70156336-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-r666f\" (UID: \"87f7ca28-e370-4e4c-85f3-818e70156336\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:12:58.756714 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.756495 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc56d\" (UniqueName: \"kubernetes.io/projected/87f7ca28-e370-4e4c-85f3-818e70156336-kube-api-access-rc56d\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-r666f\" (UID: \"87f7ca28-e370-4e4c-85f3-818e70156336\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:12:58.857614 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.857570 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f7ca28-e370-4e4c-85f3-818e70156336-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-r666f\" (UID: \"87f7ca28-e370-4e4c-85f3-818e70156336\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:12:58.857614 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.857616 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87f7ca28-e370-4e4c-85f3-818e70156336-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-r666f\" (UID: \"87f7ca28-e370-4e4c-85f3-818e70156336\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:12:58.857892 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.857657 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87f7ca28-e370-4e4c-85f3-818e70156336-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-r666f\" (UID: \"87f7ca28-e370-4e4c-85f3-818e70156336\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:12:58.857892 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.857682 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87f7ca28-e370-4e4c-85f3-818e70156336-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-r666f\" (UID: \"87f7ca28-e370-4e4c-85f3-818e70156336\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:12:58.857892 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.857728 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87f7ca28-e370-4e4c-85f3-818e70156336-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-r666f\" (UID: \"87f7ca28-e370-4e4c-85f3-818e70156336\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:12:58.857892 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.857768 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87f7ca28-e370-4e4c-85f3-818e70156336-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-r666f\" (UID: \"87f7ca28-e370-4e4c-85f3-818e70156336\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:12:58.857892 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.857817 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc56d\" (UniqueName: \"kubernetes.io/projected/87f7ca28-e370-4e4c-85f3-818e70156336-kube-api-access-rc56d\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-r666f\" (UID: \"87f7ca28-e370-4e4c-85f3-818e70156336\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:12:58.857892 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.857875 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f7ca28-e370-4e4c-85f3-818e70156336-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-r666f\" (UID: \"87f7ca28-e370-4e4c-85f3-818e70156336\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:12:58.858264 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.857908 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87f7ca28-e370-4e4c-85f3-818e70156336-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-r666f\" (UID: \"87f7ca28-e370-4e4c-85f3-818e70156336\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:12:58.858264 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.858055 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f7ca28-e370-4e4c-85f3-818e70156336-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-r666f\" (UID: \"87f7ca28-e370-4e4c-85f3-818e70156336\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:12:58.858264 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.858135 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87f7ca28-e370-4e4c-85f3-818e70156336-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-r666f\" (UID: \"87f7ca28-e370-4e4c-85f3-818e70156336\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:12:58.858417 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.858390 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f7ca28-e370-4e4c-85f3-818e70156336-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-r666f\" (UID: \"87f7ca28-e370-4e4c-85f3-818e70156336\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:12:58.858455 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.858416 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87f7ca28-e370-4e4c-85f3-818e70156336-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-r666f\" (UID: \"87f7ca28-e370-4e4c-85f3-818e70156336\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:12:58.858516 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.858492 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87f7ca28-e370-4e4c-85f3-818e70156336-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-r666f\" (UID: \"87f7ca28-e370-4e4c-85f3-818e70156336\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:12:58.860157 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.860134 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87f7ca28-e370-4e4c-85f3-818e70156336-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-r666f\" (UID: \"87f7ca28-e370-4e4c-85f3-818e70156336\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:12:58.860313 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.860296 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87f7ca28-e370-4e4c-85f3-818e70156336-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-r666f\" (UID: \"87f7ca28-e370-4e4c-85f3-818e70156336\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:12:58.920462 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.920388 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87f7ca28-e370-4e4c-85f3-818e70156336-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-r666f\" (UID: \"87f7ca28-e370-4e4c-85f3-818e70156336\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:12:58.920608 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.920470 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc56d\" (UniqueName: \"kubernetes.io/projected/87f7ca28-e370-4e4c-85f3-818e70156336-kube-api-access-rc56d\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-r666f\" (UID: \"87f7ca28-e370-4e4c-85f3-818e70156336\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:12:58.962648 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:58.962613 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:12:59.381676 ip-10-0-129-192 kubenswrapper[5615]: W1103 22:12:59.381647 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87f7ca28_e370_4e4c_85f3_818e70156336.slice/crio-60233b3651579187f80be2cd88731c5462f4931867c183d43ba80e9c7ddbc033 WatchSource:0}: Error finding container 60233b3651579187f80be2cd88731c5462f4931867c183d43ba80e9c7ddbc033: Status 404 returned error can't find the container with id 60233b3651579187f80be2cd88731c5462f4931867c183d43ba80e9c7ddbc033 Nov 03 22:12:59.383965 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:59.383928 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:12:59.384110 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:59.384008 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:12:59.384110 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:59.384052 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:12:59.384242 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:59.384143 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f"] Nov 03 22:12:59.685994 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:59.685898 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" event={"ID":"87f7ca28-e370-4e4c-85f3-818e70156336","Type":"ContainerStarted","Data":"11efbe95bfb14cb5a27a75f63a79186326a56ff6bd265c41c6d287dd85a64315"} Nov 03 22:12:59.685994 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:59.685959 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" event={"ID":"87f7ca28-e370-4e4c-85f3-818e70156336","Type":"ContainerStarted","Data":"60233b3651579187f80be2cd88731c5462f4931867c183d43ba80e9c7ddbc033"} Nov 03 22:12:59.810368 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:59.810298 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" podStartSLOduration=1.810281377 podStartE2EDuration="1.810281377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:59.808859986 +0000 UTC m=+5074.276638710" watchObservedRunningTime="2026-11-03 22:12:59.810281377 +0000 UTC m=+5074.278060100" Nov 03 22:12:59.963605 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:12:59.963501 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:13:00.965380 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:00.965345 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-r666f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 03 22:13:00.965793 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:00.965400 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" podUID="87f7ca28-e370-4e4c-85f3-818e70156336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 03 22:13:01.964042 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:01.964004 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-r666f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 03 22:13:01.964263 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:01.964070 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" podUID="87f7ca28-e370-4e4c-85f3-818e70156336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 03 22:13:02.963256 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:02.963159 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-r666f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 03 22:13:02.963256 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:02.963235 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" podUID="87f7ca28-e370-4e4c-85f3-818e70156336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 03 22:13:03.963257 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:03.963221 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-r666f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 03 22:13:03.963669 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:03.963277 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" podUID="87f7ca28-e370-4e4c-85f3-818e70156336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 03 22:13:04.963305 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:04.963262 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-r666f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 03 22:13:04.963746 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:04.963333 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" podUID="87f7ca28-e370-4e4c-85f3-818e70156336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 03 22:13:05.962961 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:05.962925 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-r666f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 03 22:13:05.963181 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:05.962981 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" podUID="87f7ca28-e370-4e4c-85f3-818e70156336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 03 22:13:06.963532 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:06.963489 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-r666f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 03 22:13:06.963944 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:06.963546 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" podUID="87f7ca28-e370-4e4c-85f3-818e70156336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 03 22:13:09.005265 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:09.005223 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-r666f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:13:09.005828 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:09.005286 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" podUID="87f7ca28-e370-4e4c-85f3-818e70156336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:13:10.047309 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:10.047259 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-r666f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:13:10.047843 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:10.047337 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" podUID="87f7ca28-e370-4e4c-85f3-818e70156336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:13:10.124820 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:10.124789 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:13:10.125069 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:10.125044 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:13:10.125882 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:10.125865 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-r666f" Nov 03 22:13:10.618270 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:13:10.618232 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:13:40.618759 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:13:40.618715 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:13:44.121982 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:44.121940 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8"] Nov 03 22:13:44.125690 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:44.125664 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8" Nov 03 22:13:44.180805 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:44.180772 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8"] Nov 03 22:13:44.213125 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:44.213064 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-afdcb5ab-epp-sa-dockercfg-ztmqr\"" Nov 03 22:13:44.213125 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:44.213067 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-afdcb5ab-kserve-self-signed-certs\"" Nov 03 22:13:44.233374 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:44.233335 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1b82464-ec18-497f-9062-e5bd331b927d-tls-certs\") pod \"e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8\" (UID: \"b1b82464-ec18-497f-9062-e5bd331b927d\") " pod="llm/e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8" Nov 03 22:13:44.233560 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:44.233487 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62j26\" (UniqueName: \"kubernetes.io/projected/b1b82464-ec18-497f-9062-e5bd331b927d-kube-api-access-62j26\") pod \"e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8\" (UID: \"b1b82464-ec18-497f-9062-e5bd331b927d\") " pod="llm/e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8" Nov 03 22:13:44.334786 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:44.334744 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62j26\" (UniqueName: \"kubernetes.io/projected/b1b82464-ec18-497f-9062-e5bd331b927d-kube-api-access-62j26\") pod \"e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8\" (UID: \"b1b82464-ec18-497f-9062-e5bd331b927d\") " pod="llm/e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8" Nov 03 22:13:44.334981 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:44.334823 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1b82464-ec18-497f-9062-e5bd331b927d-tls-certs\") pod \"e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8\" (UID: \"b1b82464-ec18-497f-9062-e5bd331b927d\") " pod="llm/e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8" Nov 03 22:13:44.337401 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:44.337380 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1b82464-ec18-497f-9062-e5bd331b927d-tls-certs\") pod \"e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8\" (UID: \"b1b82464-ec18-497f-9062-e5bd331b927d\") " pod="llm/e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8" Nov 03 22:13:44.397391 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:44.397320 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62j26\" (UniqueName: \"kubernetes.io/projected/b1b82464-ec18-497f-9062-e5bd331b927d-kube-api-access-62j26\") pod \"e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8\" (UID: \"b1b82464-ec18-497f-9062-e5bd331b927d\") " pod="llm/e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8" Nov 03 22:13:44.439310 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:44.439278 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8" Nov 03 22:13:44.807914 ip-10-0-129-192 kubenswrapper[5615]: W1103 22:13:44.807876 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b82464_ec18_497f_9062_e5bd331b927d.slice/crio-8fa988780085a7518214710294da6090dc6158663b72870bbed4f2a4ff133a4a WatchSource:0}: Error finding container 8fa988780085a7518214710294da6090dc6158663b72870bbed4f2a4ff133a4a: Status 404 returned error can't find the container with id 8fa988780085a7518214710294da6090dc6158663b72870bbed4f2a4ff133a4a Nov 03 22:13:44.809665 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:44.809590 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8"] Nov 03 22:13:45.008652 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:45.008616 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8" event={"ID":"b1b82464-ec18-497f-9062-e5bd331b927d","Type":"ContainerStarted","Data":"8fa988780085a7518214710294da6090dc6158663b72870bbed4f2a4ff133a4a"} Nov 03 22:13:47.019473 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:47.019437 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8" event={"ID":"b1b82464-ec18-497f-9062-e5bd331b927d","Type":"ContainerStarted","Data":"9bb95df76606e5818345a8c9f42489b36a99a47650e9dda276896f83db7c9c91"} Nov 03 22:13:47.019871 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:47.019566 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8" Nov 03 22:13:47.141281 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:13:47.141178 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8" podStartSLOduration=1.105392308 podStartE2EDuration="3.141158748s" totalImagesPullingTime="2.03576644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:47.140228198 +0000 UTC m=+5121.608006921" watchObservedRunningTime="2026-11-03 22:13:47.141158748 +0000 UTC m=+5121.608937470" Nov 03 22:14:10.618340 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:14:10.618259 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:14:18.028035 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:14:18.028002 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8" Nov 03 22:14:40.618514 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:14:40.618475 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:14:50.707139 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:14:50.707101 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8"] Nov 03 22:14:58.049102 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:14:58.048191 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8" podUID="b1b82464-ec18-497f-9062-e5bd331b927d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:15:05.728181 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:05.728120 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8" podUID="b1b82464-ec18-497f-9062-e5bd331b927d" containerName="main" containerID="cri-o://9bb95df76606e5818345a8c9f42489b36a99a47650e9dda276896f83db7c9c91" gracePeriod=45 Nov 03 22:15:08.028206 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:08.028160 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8" podUID="b1b82464-ec18-497f-9062-e5bd331b927d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:15:10.618675 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:15:10.618632 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:15:18.028701 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:18.028653 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8" podUID="b1b82464-ec18-497f-9062-e5bd331b927d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:15:18.029199 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:18.028790 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8" Nov 03 22:15:28.028258 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:28.028215 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8" podUID="b1b82464-ec18-497f-9062-e5bd331b927d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:15:35.983131 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:35.983103 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8" Nov 03 22:15:36.019430 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:36.019345 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1b82464-ec18-497f-9062-e5bd331b927d-kube-api-access-62j26\" (UniqueName: \"kubernetes.io/projected/b1b82464-ec18-497f-9062-e5bd331b927d-kube-api-access-62j26\") pod \"b1b82464-ec18-497f-9062-e5bd331b927d\" (UID: \"b1b82464-ec18-497f-9062-e5bd331b927d\") " Nov 03 22:15:36.019430 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:36.019418 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1b82464-ec18-497f-9062-e5bd331b927d-tls-certs\" (UniqueName: \"kubernetes.io/secret/b1b82464-ec18-497f-9062-e5bd331b927d-tls-certs\") pod \"b1b82464-ec18-497f-9062-e5bd331b927d\" (UID: \"b1b82464-ec18-497f-9062-e5bd331b927d\") " Nov 03 22:15:36.021640 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:36.021609 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b82464-ec18-497f-9062-e5bd331b927d-kube-api-access-62j26" pod "b1b82464-ec18-497f-9062-e5bd331b927d" (UID: "b1b82464-ec18-497f-9062-e5bd331b927d"). InnerVolumeSpecName "kube-api-access-62j26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:15:36.021752 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:36.021668 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1b82464-ec18-497f-9062-e5bd331b927d-tls-certs" pod "b1b82464-ec18-497f-9062-e5bd331b927d" (UID: "b1b82464-ec18-497f-9062-e5bd331b927d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:15:36.120728 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:36.120692 5615 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1b82464-ec18-497f-9062-e5bd331b927d-tls-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:15:36.120728 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:36.120726 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62j26\" (UniqueName: \"kubernetes.io/projected/b1b82464-ec18-497f-9062-e5bd331b927d-kube-api-access-62j26\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:15:36.757279 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:36.757245 5615 generic.go:356] "Generic (PLEG): container finished" podID="b1b82464-ec18-497f-9062-e5bd331b927d" containerID="9bb95df76606e5818345a8c9f42489b36a99a47650e9dda276896f83db7c9c91" exitCode=0 Nov 03 22:15:36.757478 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:36.757297 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8" Nov 03 22:15:36.757478 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:36.757336 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8" event={"ID":"b1b82464-ec18-497f-9062-e5bd331b927d","Type":"ContainerDied","Data":"9bb95df76606e5818345a8c9f42489b36a99a47650e9dda276896f83db7c9c91"} Nov 03 22:15:36.757478 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:36.757389 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8" event={"ID":"b1b82464-ec18-497f-9062-e5bd331b927d","Type":"ContainerDied","Data":"8fa988780085a7518214710294da6090dc6158663b72870bbed4f2a4ff133a4a"} Nov 03 22:15:36.757478 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:36.757411 5615 scope.go:122] "RemoveContainer" containerID="9bb95df76606e5818345a8c9f42489b36a99a47650e9dda276896f83db7c9c91" Nov 03 22:15:36.769118 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:36.768912 5615 scope.go:122] "RemoveContainer" containerID="9bb95df76606e5818345a8c9f42489b36a99a47650e9dda276896f83db7c9c91" Nov 03 22:15:36.769336 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:15:36.769311 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bb95df76606e5818345a8c9f42489b36a99a47650e9dda276896f83db7c9c91\": container with ID starting with 9bb95df76606e5818345a8c9f42489b36a99a47650e9dda276896f83db7c9c91 not found: ID does not exist" containerID="9bb95df76606e5818345a8c9f42489b36a99a47650e9dda276896f83db7c9c91" Nov 03 22:15:36.769404 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:36.769350 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bb95df76606e5818345a8c9f42489b36a99a47650e9dda276896f83db7c9c91"} err="failed to get container status \"9bb95df76606e5818345a8c9f42489b36a99a47650e9dda276896f83db7c9c91\": rpc error: code = NotFound desc = could not find container \"9bb95df76606e5818345a8c9f42489b36a99a47650e9dda276896f83db7c9c91\": container with ID starting with 9bb95df76606e5818345a8c9f42489b36a99a47650e9dda276896f83db7c9c91 not found: ID does not exist" Nov 03 22:15:36.925717 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:36.925683 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8"] Nov 03 22:15:36.977586 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:36.977538 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-afdcb5ab-kserve-router-scheduler-7d4bfcb6cd-rtln8"] Nov 03 22:15:37.651488 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:37.651455 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1b82464-ec18-497f-9062-e5bd331b927d" path="/var/lib/kubelet/pods/b1b82464-ec18-497f-9062-e5bd331b927d/volumes" Nov 03 22:15:40.619173 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:15:40.619136 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:15:42.515325 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:42.515287 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs"] Nov 03 22:15:42.515762 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:42.515747 5615 cpu_manager.go:414] "containerMap: removing container" podUID="b1b82464-ec18-497f-9062-e5bd331b927d" containerName="main" Nov 03 22:15:42.515807 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:42.515762 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b82464-ec18-497f-9062-e5bd331b927d" containerName="main" Nov 03 22:15:42.515848 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:42.515835 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b82464-ec18-497f-9062-e5bd331b927d" containerName="main" Nov 03 22:15:42.518809 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:42.518791 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" Nov 03 22:15:42.572488 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:42.572455 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-60736a\"/\"openshift-service-ca.crt\"" Nov 03 22:15:42.587302 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:42.587259 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb1c84a9-d1af-438d-8a08-126865b81cee-tmp-dir\") pod \"simulator-single-main-60736a-kserve-b487f69ff-9p5vs\" (UID: \"bb1c84a9-d1af-438d-8a08-126865b81cee\") " pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" Nov 03 22:15:42.587495 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:42.587337 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb1c84a9-d1af-438d-8a08-126865b81cee-dshm\") pod \"simulator-single-main-60736a-kserve-b487f69ff-9p5vs\" (UID: \"bb1c84a9-d1af-438d-8a08-126865b81cee\") " pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" Nov 03 22:15:42.587495 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:42.587439 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb1c84a9-d1af-438d-8a08-126865b81cee-home\") pod \"simulator-single-main-60736a-kserve-b487f69ff-9p5vs\" (UID: \"bb1c84a9-d1af-438d-8a08-126865b81cee\") " pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" Nov 03 22:15:42.587593 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:42.587493 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntpzt\" (UniqueName: \"kubernetes.io/projected/bb1c84a9-d1af-438d-8a08-126865b81cee-kube-api-access-ntpzt\") pod \"simulator-single-main-60736a-kserve-b487f69ff-9p5vs\" (UID: \"bb1c84a9-d1af-438d-8a08-126865b81cee\") " pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" Nov 03 22:15:42.587638 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:42.587583 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb1c84a9-d1af-438d-8a08-126865b81cee-model-cache\") pod \"simulator-single-main-60736a-kserve-b487f69ff-9p5vs\" (UID: \"bb1c84a9-d1af-438d-8a08-126865b81cee\") " pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" Nov 03 22:15:42.587638 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:42.587611 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb1c84a9-d1af-438d-8a08-126865b81cee-tls-certs\") pod \"simulator-single-main-60736a-kserve-b487f69ff-9p5vs\" (UID: \"bb1c84a9-d1af-438d-8a08-126865b81cee\") " pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" Nov 03 22:15:42.623523 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:42.623483 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-60736a\"/\"kube-root-ca.crt\"" Nov 03 22:15:42.623889 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:42.623866 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-60736a\"/\"default-dockercfg-dwkrc\"" Nov 03 22:15:42.624015 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:42.623966 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-60736a\"/\"simulator-single-main-60736a-kserve-self-signed-certs\"" Nov 03 22:15:42.625372 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:42.625348 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs"] Nov 03 22:15:42.690525 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:42.690479 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb1c84a9-d1af-438d-8a08-126865b81cee-tmp-dir\") pod \"simulator-single-main-60736a-kserve-b487f69ff-9p5vs\" (UID: \"bb1c84a9-d1af-438d-8a08-126865b81cee\") " pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" Nov 03 22:15:42.690741 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:42.690645 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb1c84a9-d1af-438d-8a08-126865b81cee-dshm\") pod \"simulator-single-main-60736a-kserve-b487f69ff-9p5vs\" (UID: \"bb1c84a9-d1af-438d-8a08-126865b81cee\") " pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" Nov 03 22:15:42.690741 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:42.690693 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb1c84a9-d1af-438d-8a08-126865b81cee-home\") pod \"simulator-single-main-60736a-kserve-b487f69ff-9p5vs\" (UID: \"bb1c84a9-d1af-438d-8a08-126865b81cee\") " pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" Nov 03 22:15:42.690947 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:42.690748 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntpzt\" (UniqueName: \"kubernetes.io/projected/bb1c84a9-d1af-438d-8a08-126865b81cee-kube-api-access-ntpzt\") pod \"simulator-single-main-60736a-kserve-b487f69ff-9p5vs\" (UID: \"bb1c84a9-d1af-438d-8a08-126865b81cee\") " pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" Nov 03 22:15:42.690947 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:42.690800 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb1c84a9-d1af-438d-8a08-126865b81cee-model-cache\") pod \"simulator-single-main-60736a-kserve-b487f69ff-9p5vs\" (UID: \"bb1c84a9-d1af-438d-8a08-126865b81cee\") " pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" Nov 03 22:15:42.690947 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:42.690824 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb1c84a9-d1af-438d-8a08-126865b81cee-tls-certs\") pod \"simulator-single-main-60736a-kserve-b487f69ff-9p5vs\" (UID: \"bb1c84a9-d1af-438d-8a08-126865b81cee\") " pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" Nov 03 22:15:42.690947 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:42.690873 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb1c84a9-d1af-438d-8a08-126865b81cee-tmp-dir\") pod \"simulator-single-main-60736a-kserve-b487f69ff-9p5vs\" (UID: \"bb1c84a9-d1af-438d-8a08-126865b81cee\") " pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" Nov 03 22:15:42.691333 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:42.691150 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb1c84a9-d1af-438d-8a08-126865b81cee-home\") pod \"simulator-single-main-60736a-kserve-b487f69ff-9p5vs\" (UID: \"bb1c84a9-d1af-438d-8a08-126865b81cee\") " pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" Nov 03 22:15:42.691683 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:42.691658 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb1c84a9-d1af-438d-8a08-126865b81cee-model-cache\") pod \"simulator-single-main-60736a-kserve-b487f69ff-9p5vs\" (UID: \"bb1c84a9-d1af-438d-8a08-126865b81cee\") " pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" Nov 03 22:15:42.693795 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:42.693769 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb1c84a9-d1af-438d-8a08-126865b81cee-dshm\") pod \"simulator-single-main-60736a-kserve-b487f69ff-9p5vs\" (UID: \"bb1c84a9-d1af-438d-8a08-126865b81cee\") " pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" Nov 03 22:15:42.693889 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:42.693864 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb1c84a9-d1af-438d-8a08-126865b81cee-tls-certs\") pod \"simulator-single-main-60736a-kserve-b487f69ff-9p5vs\" (UID: \"bb1c84a9-d1af-438d-8a08-126865b81cee\") " pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" Nov 03 22:15:42.753496 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:42.753457 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntpzt\" (UniqueName: \"kubernetes.io/projected/bb1c84a9-d1af-438d-8a08-126865b81cee-kube-api-access-ntpzt\") pod \"simulator-single-main-60736a-kserve-b487f69ff-9p5vs\" (UID: \"bb1c84a9-d1af-438d-8a08-126865b81cee\") " pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" Nov 03 22:15:42.831331 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:42.831236 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" Nov 03 22:15:43.205202 ip-10-0-129-192 kubenswrapper[5615]: W1103 22:15:43.205174 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb1c84a9_d1af_438d_8a08_126865b81cee.slice/crio-d7a6f99139af347df75a6e4c0010272e16ad08583a6dc332cc3643944dd1723d WatchSource:0}: Error finding container d7a6f99139af347df75a6e4c0010272e16ad08583a6dc332cc3643944dd1723d: Status 404 returned error can't find the container with id d7a6f99139af347df75a6e4c0010272e16ad08583a6dc332cc3643944dd1723d Nov 03 22:15:43.206464 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:43.206442 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs"] Nov 03 22:15:43.797421 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:43.797387 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" event={"ID":"bb1c84a9-d1af-438d-8a08-126865b81cee","Type":"ContainerStarted","Data":"7cd6221163cc550599b8bc351e4a712be24e7cc5a912ffa1e3a99e9dd869a3da"} Nov 03 22:15:43.797421 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:43.797427 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" event={"ID":"bb1c84a9-d1af-438d-8a08-126865b81cee","Type":"ContainerStarted","Data":"d7a6f99139af347df75a6e4c0010272e16ad08583a6dc332cc3643944dd1723d"} Nov 03 22:15:43.831895 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:43.831859 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" Nov 03 22:15:43.917687 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:43.917626 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" podStartSLOduration=1.917611216 podStartE2EDuration="1.917611216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:43.916453254 +0000 UTC m=+5238.384231976" watchObservedRunningTime="2026-11-03 22:15:43.917611216 +0000 UTC m=+5238.385389916" Nov 03 22:15:52.831902 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:52.831859 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" Nov 03 22:15:52.844428 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:52.844398 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" Nov 03 22:15:52.868482 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:15:52.868453 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" Nov 03 22:16:10.619531 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:16:10.619488 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:16:38.910644 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:16:38.910606 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs"] Nov 03 22:16:40.618683 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:16:40.618639 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:16:53.931147 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:16:53.931101 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" podUID="bb1c84a9-d1af-438d-8a08-126865b81cee" containerName="main" containerID="cri-o://7cd6221163cc550599b8bc351e4a712be24e7cc5a912ffa1e3a99e9dd869a3da" gracePeriod=45 Nov 03 22:16:54.172916 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:16:54.172880 5615 generic.go:356] "Generic (PLEG): container finished" podID="bb1c84a9-d1af-438d-8a08-126865b81cee" containerID="7cd6221163cc550599b8bc351e4a712be24e7cc5a912ffa1e3a99e9dd869a3da" exitCode=0 Nov 03 22:16:54.173125 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:16:54.172954 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" event={"ID":"bb1c84a9-d1af-438d-8a08-126865b81cee","Type":"ContainerDied","Data":"7cd6221163cc550599b8bc351e4a712be24e7cc5a912ffa1e3a99e9dd869a3da"} Nov 03 22:16:54.173125 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:16:54.172993 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" event={"ID":"bb1c84a9-d1af-438d-8a08-126865b81cee","Type":"ContainerDied","Data":"d7a6f99139af347df75a6e4c0010272e16ad08583a6dc332cc3643944dd1723d"} Nov 03 22:16:54.173125 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:16:54.173005 5615 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7a6f99139af347df75a6e4c0010272e16ad08583a6dc332cc3643944dd1723d" Nov 03 22:16:54.181524 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:16:54.181460 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" Nov 03 22:16:54.261893 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:16:54.261856 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb1c84a9-d1af-438d-8a08-126865b81cee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb1c84a9-d1af-438d-8a08-126865b81cee-model-cache\") pod \"bb1c84a9-d1af-438d-8a08-126865b81cee\" (UID: \"bb1c84a9-d1af-438d-8a08-126865b81cee\") " Nov 03 22:16:54.261893 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:16:54.261897 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb1c84a9-d1af-438d-8a08-126865b81cee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb1c84a9-d1af-438d-8a08-126865b81cee-dshm\") pod \"bb1c84a9-d1af-438d-8a08-126865b81cee\" (UID: \"bb1c84a9-d1af-438d-8a08-126865b81cee\") " Nov 03 22:16:54.262175 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:16:54.261929 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb1c84a9-d1af-438d-8a08-126865b81cee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb1c84a9-d1af-438d-8a08-126865b81cee-tmp-dir\") pod \"bb1c84a9-d1af-438d-8a08-126865b81cee\" (UID: \"bb1c84a9-d1af-438d-8a08-126865b81cee\") " Nov 03 22:16:54.262175 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:16:54.261973 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb1c84a9-d1af-438d-8a08-126865b81cee-tls-certs\" (UniqueName: \"kubernetes.io/secret/bb1c84a9-d1af-438d-8a08-126865b81cee-tls-certs\") pod \"bb1c84a9-d1af-438d-8a08-126865b81cee\" (UID: \"bb1c84a9-d1af-438d-8a08-126865b81cee\") " Nov 03 22:16:54.262175 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:16:54.262009 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb1c84a9-d1af-438d-8a08-126865b81cee-home\" (UniqueName: \"kubernetes.io/empty-dir/bb1c84a9-d1af-438d-8a08-126865b81cee-home\") pod \"bb1c84a9-d1af-438d-8a08-126865b81cee\" (UID: \"bb1c84a9-d1af-438d-8a08-126865b81cee\") " Nov 03 22:16:54.262175 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:16:54.262115 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb1c84a9-d1af-438d-8a08-126865b81cee-kube-api-access-ntpzt\" (UniqueName: \"kubernetes.io/projected/bb1c84a9-d1af-438d-8a08-126865b81cee-kube-api-access-ntpzt\") pod \"bb1c84a9-d1af-438d-8a08-126865b81cee\" (UID: \"bb1c84a9-d1af-438d-8a08-126865b81cee\") " Nov 03 22:16:54.262411 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:16:54.262195 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1c84a9-d1af-438d-8a08-126865b81cee-model-cache" pod "bb1c84a9-d1af-438d-8a08-126865b81cee" (UID: "bb1c84a9-d1af-438d-8a08-126865b81cee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:16:54.262411 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:16:54.262262 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1c84a9-d1af-438d-8a08-126865b81cee-tmp-dir" pod "bb1c84a9-d1af-438d-8a08-126865b81cee" (UID: "bb1c84a9-d1af-438d-8a08-126865b81cee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:16:54.262411 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:16:54.262380 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1c84a9-d1af-438d-8a08-126865b81cee-home" pod "bb1c84a9-d1af-438d-8a08-126865b81cee" (UID: "bb1c84a9-d1af-438d-8a08-126865b81cee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:16:54.262567 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:16:54.262512 5615 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb1c84a9-d1af-438d-8a08-126865b81cee-model-cache\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:16:54.262567 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:16:54.262534 5615 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb1c84a9-d1af-438d-8a08-126865b81cee-tmp-dir\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:16:54.262567 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:16:54.262550 5615 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb1c84a9-d1af-438d-8a08-126865b81cee-home\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:16:54.264260 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:16:54.264236 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1c84a9-d1af-438d-8a08-126865b81cee-dshm" pod "bb1c84a9-d1af-438d-8a08-126865b81cee" (UID: "bb1c84a9-d1af-438d-8a08-126865b81cee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:16:54.264378 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:16:54.264296 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb1c84a9-d1af-438d-8a08-126865b81cee-tls-certs" pod "bb1c84a9-d1af-438d-8a08-126865b81cee" (UID: "bb1c84a9-d1af-438d-8a08-126865b81cee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:16:54.264378 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:16:54.264337 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb1c84a9-d1af-438d-8a08-126865b81cee-kube-api-access-ntpzt" pod "bb1c84a9-d1af-438d-8a08-126865b81cee" (UID: "bb1c84a9-d1af-438d-8a08-126865b81cee"). InnerVolumeSpecName "kube-api-access-ntpzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:16:54.363942 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:16:54.363904 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntpzt\" (UniqueName: \"kubernetes.io/projected/bb1c84a9-d1af-438d-8a08-126865b81cee-kube-api-access-ntpzt\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:16:54.363942 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:16:54.363941 5615 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb1c84a9-d1af-438d-8a08-126865b81cee-dshm\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:16:54.364195 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:16:54.363955 5615 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb1c84a9-d1af-438d-8a08-126865b81cee-tls-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:16:55.178954 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:16:55.178910 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs" Nov 03 22:16:55.347274 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:16:55.347230 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs"] Nov 03 22:16:55.352925 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:16:55.352899 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-60736a/simulator-single-main-60736a-kserve-b487f69ff-9p5vs"] Nov 03 22:16:55.650508 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:16:55.650473 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb1c84a9-d1af-438d-8a08-126865b81cee" path="/var/lib/kubelet/pods/bb1c84a9-d1af-438d-8a08-126865b81cee/volumes" Nov 03 22:17:10.619473 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:17:10.619381 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:17:17.939096 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:17.939038 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6"] Nov 03 22:17:17.939515 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:17.939489 5615 cpu_manager.go:414] "containerMap: removing container" podUID="bb1c84a9-d1af-438d-8a08-126865b81cee" containerName="main" Nov 03 22:17:17.939515 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:17.939501 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb1c84a9-d1af-438d-8a08-126865b81cee" containerName="main" Nov 03 22:17:17.939609 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:17.939598 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb1c84a9-d1af-438d-8a08-126865b81cee" containerName="main" Nov 03 22:17:17.942764 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:17.942740 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6" Nov 03 22:17:18.012975 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:18.012912 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncxhq\" (UniqueName: \"kubernetes.io/projected/0469d23e-8be1-4425-946c-9c8f9cff3792-kube-api-access-ncxhq\") pod \"maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6\" (UID: \"0469d23e-8be1-4425-946c-9c8f9cff3792\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6" Nov 03 22:17:18.013187 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:18.013039 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0469d23e-8be1-4425-946c-9c8f9cff3792-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6\" (UID: \"0469d23e-8be1-4425-946c-9c8f9cff3792\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6" Nov 03 22:17:18.013187 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:18.013141 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0469d23e-8be1-4425-946c-9c8f9cff3792-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6\" (UID: \"0469d23e-8be1-4425-946c-9c8f9cff3792\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6" Nov 03 22:17:18.013297 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:18.013250 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0469d23e-8be1-4425-946c-9c8f9cff3792-metrics-certs\") pod \"maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6\" (UID: \"0469d23e-8be1-4425-946c-9c8f9cff3792\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6" Nov 03 22:17:18.050518 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:18.050482 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6"] Nov 03 22:17:18.066380 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:18.066348 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-60736a-serving-cert\"" Nov 03 22:17:18.114222 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:18.114186 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncxhq\" (UniqueName: \"kubernetes.io/projected/0469d23e-8be1-4425-946c-9c8f9cff3792-kube-api-access-ncxhq\") pod \"maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6\" (UID: \"0469d23e-8be1-4425-946c-9c8f9cff3792\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6" Nov 03 22:17:18.114423 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:18.114237 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0469d23e-8be1-4425-946c-9c8f9cff3792-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6\" (UID: \"0469d23e-8be1-4425-946c-9c8f9cff3792\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6" Nov 03 22:17:18.114423 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:18.114305 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0469d23e-8be1-4425-946c-9c8f9cff3792-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6\" (UID: \"0469d23e-8be1-4425-946c-9c8f9cff3792\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6" Nov 03 22:17:18.114423 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:18.114351 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0469d23e-8be1-4425-946c-9c8f9cff3792-metrics-certs\") pod \"maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6\" (UID: \"0469d23e-8be1-4425-946c-9c8f9cff3792\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6" Nov 03 22:17:18.114614 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:17:18.114458 5615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-serving-cert: secret "maas-api-e2e-worker-single-main-60736a-serving-cert" not found Nov 03 22:17:18.114614 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:17:18.114521 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0469d23e-8be1-4425-946c-9c8f9cff3792-maas-api-tls podName:0469d23e-8be1-4425-946c-9c8f9cff3792 nodeName:}" failed. No retries permitted until 2026-11-03 22:17:18.614504798 +0000 UTC m=+5333.082283512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0469d23e-8be1-4425-946c-9c8f9cff3792-maas-api-tls") pod "maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6" (UID: "0469d23e-8be1-4425-946c-9c8f9cff3792") : secret "maas-api-e2e-worker-single-main-60736a-serving-cert" not found Nov 03 22:17:18.114950 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:18.114927 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0469d23e-8be1-4425-946c-9c8f9cff3792-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6\" (UID: \"0469d23e-8be1-4425-946c-9c8f9cff3792\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6" Nov 03 22:17:18.116827 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:18.116803 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0469d23e-8be1-4425-946c-9c8f9cff3792-metrics-certs\") pod \"maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6\" (UID: \"0469d23e-8be1-4425-946c-9c8f9cff3792\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6" Nov 03 22:17:18.177030 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:18.176993 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncxhq\" (UniqueName: \"kubernetes.io/projected/0469d23e-8be1-4425-946c-9c8f9cff3792-kube-api-access-ncxhq\") pod \"maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6\" (UID: \"0469d23e-8be1-4425-946c-9c8f9cff3792\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6" Nov 03 22:17:18.621523 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:18.621481 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0469d23e-8be1-4425-946c-9c8f9cff3792-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6\" (UID: \"0469d23e-8be1-4425-946c-9c8f9cff3792\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6" Nov 03 22:17:18.623958 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:18.623937 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0469d23e-8be1-4425-946c-9c8f9cff3792-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6\" (UID: \"0469d23e-8be1-4425-946c-9c8f9cff3792\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6" Nov 03 22:17:18.856874 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:18.856834 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6" Nov 03 22:17:19.233956 ip-10-0-129-192 kubenswrapper[5615]: W1103 22:17:19.233911 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0469d23e_8be1_4425_946c_9c8f9cff3792.slice/crio-adb79a49fc9b4d10cedbbed23593861132f8fa509b78bbbbce125325e3f72167 WatchSource:0}: Error finding container adb79a49fc9b4d10cedbbed23593861132f8fa509b78bbbbce125325e3f72167: Status 404 returned error can't find the container with id adb79a49fc9b4d10cedbbed23593861132f8fa509b78bbbbce125325e3f72167 Nov 03 22:17:19.235422 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:19.235393 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6"] Nov 03 22:17:19.235835 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:19.235814 5615 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:17:19.296510 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:19.296472 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6" event={"ID":"0469d23e-8be1-4425-946c-9c8f9cff3792","Type":"ContainerStarted","Data":"adb79a49fc9b4d10cedbbed23593861132f8fa509b78bbbbce125325e3f72167"} Nov 03 22:17:20.302712 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:20.302677 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6" event={"ID":"0469d23e-8be1-4425-946c-9c8f9cff3792","Type":"ContainerStarted","Data":"823c25f70c8dd0608f1b86499d57b881927983c94bf43390d2acb7453fe9eda5"} Nov 03 22:17:20.303219 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:20.302804 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6" Nov 03 22:17:20.434788 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:20.434684 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6" podStartSLOduration=2.523751749 podStartE2EDuration="3.434668745s" totalImagesPullingTime="910.916996ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:20.434437543 +0000 UTC m=+5334.902216268" watchObservedRunningTime="2026-11-03 22:17:20.434668745 +0000 UTC m=+5334.902447467" Nov 03 22:17:26.313523 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:26.313494 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6" Nov 03 22:17:40.618927 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:17:40.618887 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:17:47.521224 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:47.521184 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6"] Nov 03 22:17:47.521790 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:47.521415 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6" podUID="0469d23e-8be1-4425-946c-9c8f9cff3792" containerName="maas-api" containerID="cri-o://823c25f70c8dd0608f1b86499d57b881927983c94bf43390d2acb7453fe9eda5" gracePeriod=30 Nov 03 22:17:47.789164 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:47.789131 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6" Nov 03 22:17:47.853308 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:47.853267 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0469d23e-8be1-4425-946c-9c8f9cff3792-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0469d23e-8be1-4425-946c-9c8f9cff3792-openshift-service-ca\") pod \"0469d23e-8be1-4425-946c-9c8f9cff3792\" (UID: \"0469d23e-8be1-4425-946c-9c8f9cff3792\") " Nov 03 22:17:47.853525 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:47.853347 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0469d23e-8be1-4425-946c-9c8f9cff3792-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0469d23e-8be1-4425-946c-9c8f9cff3792-metrics-certs\") pod \"0469d23e-8be1-4425-946c-9c8f9cff3792\" (UID: \"0469d23e-8be1-4425-946c-9c8f9cff3792\") " Nov 03 22:17:47.853525 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:47.853379 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0469d23e-8be1-4425-946c-9c8f9cff3792-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0469d23e-8be1-4425-946c-9c8f9cff3792-maas-api-tls\") pod \"0469d23e-8be1-4425-946c-9c8f9cff3792\" (UID: \"0469d23e-8be1-4425-946c-9c8f9cff3792\") " Nov 03 22:17:47.853525 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:47.853422 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0469d23e-8be1-4425-946c-9c8f9cff3792-kube-api-access-ncxhq\" (UniqueName: \"kubernetes.io/projected/0469d23e-8be1-4425-946c-9c8f9cff3792-kube-api-access-ncxhq\") pod \"0469d23e-8be1-4425-946c-9c8f9cff3792\" (UID: \"0469d23e-8be1-4425-946c-9c8f9cff3792\") " Nov 03 22:17:47.853706 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:47.853662 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0469d23e-8be1-4425-946c-9c8f9cff3792-openshift-service-ca" pod "0469d23e-8be1-4425-946c-9c8f9cff3792" (UID: "0469d23e-8be1-4425-946c-9c8f9cff3792"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:17:47.853784 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:47.853757 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0469d23e-8be1-4425-946c-9c8f9cff3792-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:17:47.855601 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:47.855577 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0469d23e-8be1-4425-946c-9c8f9cff3792-metrics-certs" pod "0469d23e-8be1-4425-946c-9c8f9cff3792" (UID: "0469d23e-8be1-4425-946c-9c8f9cff3792"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:17:47.855721 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:47.855650 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0469d23e-8be1-4425-946c-9c8f9cff3792-maas-api-tls" pod "0469d23e-8be1-4425-946c-9c8f9cff3792" (UID: "0469d23e-8be1-4425-946c-9c8f9cff3792"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:17:47.855770 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:47.855740 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0469d23e-8be1-4425-946c-9c8f9cff3792-kube-api-access-ncxhq" pod "0469d23e-8be1-4425-946c-9c8f9cff3792" (UID: "0469d23e-8be1-4425-946c-9c8f9cff3792"). InnerVolumeSpecName "kube-api-access-ncxhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:17:47.954747 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:47.954698 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0469d23e-8be1-4425-946c-9c8f9cff3792-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:17:47.954747 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:47.954744 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0469d23e-8be1-4425-946c-9c8f9cff3792-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:17:47.954980 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:47.954778 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncxhq\" (UniqueName: \"kubernetes.io/projected/0469d23e-8be1-4425-946c-9c8f9cff3792-kube-api-access-ncxhq\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:17:48.450935 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:48.450898 5615 generic.go:356] "Generic (PLEG): container finished" podID="0469d23e-8be1-4425-946c-9c8f9cff3792" containerID="823c25f70c8dd0608f1b86499d57b881927983c94bf43390d2acb7453fe9eda5" exitCode=0 Nov 03 22:17:48.451159 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:48.450946 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6" Nov 03 22:17:48.451159 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:48.450973 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6" event={"ID":"0469d23e-8be1-4425-946c-9c8f9cff3792","Type":"ContainerDied","Data":"823c25f70c8dd0608f1b86499d57b881927983c94bf43390d2acb7453fe9eda5"} Nov 03 22:17:48.451159 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:48.451029 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6" event={"ID":"0469d23e-8be1-4425-946c-9c8f9cff3792","Type":"ContainerDied","Data":"adb79a49fc9b4d10cedbbed23593861132f8fa509b78bbbbce125325e3f72167"} Nov 03 22:17:48.451159 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:48.451051 5615 scope.go:122] "RemoveContainer" containerID="823c25f70c8dd0608f1b86499d57b881927983c94bf43390d2acb7453fe9eda5" Nov 03 22:17:48.460881 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:48.460862 5615 scope.go:122] "RemoveContainer" containerID="823c25f70c8dd0608f1b86499d57b881927983c94bf43390d2acb7453fe9eda5" Nov 03 22:17:48.461186 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:17:48.461162 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"823c25f70c8dd0608f1b86499d57b881927983c94bf43390d2acb7453fe9eda5\": container with ID starting with 823c25f70c8dd0608f1b86499d57b881927983c94bf43390d2acb7453fe9eda5 not found: ID does not exist" containerID="823c25f70c8dd0608f1b86499d57b881927983c94bf43390d2acb7453fe9eda5" Nov 03 22:17:48.461275 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:48.461197 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"823c25f70c8dd0608f1b86499d57b881927983c94bf43390d2acb7453fe9eda5"} err="failed to get container status \"823c25f70c8dd0608f1b86499d57b881927983c94bf43390d2acb7453fe9eda5\": rpc error: code = NotFound desc = could not find container \"823c25f70c8dd0608f1b86499d57b881927983c94bf43390d2acb7453fe9eda5\": container with ID starting with 823c25f70c8dd0608f1b86499d57b881927983c94bf43390d2acb7453fe9eda5 not found: ID does not exist" Nov 03 22:17:48.619165 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:48.619127 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6"] Nov 03 22:17:48.622610 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:48.622586 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-9t5k6"] Nov 03 22:17:49.650921 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:17:49.650889 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0469d23e-8be1-4425-946c-9c8f9cff3792" path="/var/lib/kubelet/pods/0469d23e-8be1-4425-946c-9c8f9cff3792/volumes" Nov 03 22:18:10.618719 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:18:10.618679 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:18:40.619097 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:18:40.618993 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:19:10.617992 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:19:10.617950 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:19:40.618527 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:19:40.618481 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:19:45.614232 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:19:45.614183 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:19:45.614913 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:19:45.614875 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" podUID="cb73420b-7000-4ae1-99de-e72bdd6b847b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:20:10.617510 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:20:10.617427 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:20:25.261230 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:25.257905 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-jjmps"] Nov 03 22:20:30.258642 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:30.258590 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jjmps" podUID="2f60591a-fc5a-47d9-9899-93963284b5d6" containerName="payload-pre-processing" containerID="cri-o://624269bf12c8e3fc860c198e1010d5e2052e217f400ee3639b17f25083ff5fa4" gracePeriod=25 Nov 03 22:20:30.503282 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:30.503250 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jjmps" Nov 03 22:20:30.534764 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:30.534668 5615 generic.go:356] "Generic (PLEG): container finished" podID="2f60591a-fc5a-47d9-9899-93963284b5d6" containerID="624269bf12c8e3fc860c198e1010d5e2052e217f400ee3639b17f25083ff5fa4" exitCode=0 Nov 03 22:20:30.534764 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:30.534710 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jjmps" Nov 03 22:20:30.534764 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:30.534727 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jjmps" event={"ID":"2f60591a-fc5a-47d9-9899-93963284b5d6","Type":"ContainerDied","Data":"624269bf12c8e3fc860c198e1010d5e2052e217f400ee3639b17f25083ff5fa4"} Nov 03 22:20:30.535013 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:30.534794 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jjmps" event={"ID":"2f60591a-fc5a-47d9-9899-93963284b5d6","Type":"ContainerDied","Data":"4aea7f5951c5b0b095e698a901b31bb19b80ce35f61613e09b5a174dd543471b"} Nov 03 22:20:30.535013 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:30.534817 5615 scope.go:122] "RemoveContainer" containerID="624269bf12c8e3fc860c198e1010d5e2052e217f400ee3639b17f25083ff5fa4" Nov 03 22:20:30.543429 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:30.543403 5615 scope.go:122] "RemoveContainer" containerID="624269bf12c8e3fc860c198e1010d5e2052e217f400ee3639b17f25083ff5fa4" Nov 03 22:20:30.543711 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:20:30.543690 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"624269bf12c8e3fc860c198e1010d5e2052e217f400ee3639b17f25083ff5fa4\": container with ID starting with 624269bf12c8e3fc860c198e1010d5e2052e217f400ee3639b17f25083ff5fa4 not found: ID does not exist" containerID="624269bf12c8e3fc860c198e1010d5e2052e217f400ee3639b17f25083ff5fa4" Nov 03 22:20:30.543769 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:30.543725 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"624269bf12c8e3fc860c198e1010d5e2052e217f400ee3639b17f25083ff5fa4"} err="failed to get container status \"624269bf12c8e3fc860c198e1010d5e2052e217f400ee3639b17f25083ff5fa4\": rpc error: code = NotFound desc = could not find container \"624269bf12c8e3fc860c198e1010d5e2052e217f400ee3639b17f25083ff5fa4\": container with ID starting with 624269bf12c8e3fc860c198e1010d5e2052e217f400ee3639b17f25083ff5fa4 not found: ID does not exist" Nov 03 22:20:30.607047 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:30.607010 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f60591a-fc5a-47d9-9899-93963284b5d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f60591a-fc5a-47d9-9899-93963284b5d6-plugins-config-volume\") pod \"2f60591a-fc5a-47d9-9899-93963284b5d6\" (UID: \"2f60591a-fc5a-47d9-9899-93963284b5d6\") " Nov 03 22:20:30.607250 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:30.607181 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f60591a-fc5a-47d9-9899-93963284b5d6-kube-api-access-vkbml\" (UniqueName: \"kubernetes.io/projected/2f60591a-fc5a-47d9-9899-93963284b5d6-kube-api-access-vkbml\") pod \"2f60591a-fc5a-47d9-9899-93963284b5d6\" (UID: \"2f60591a-fc5a-47d9-9899-93963284b5d6\") " Nov 03 22:20:30.607408 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:30.607384 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f60591a-fc5a-47d9-9899-93963284b5d6-plugins-config-volume" pod "2f60591a-fc5a-47d9-9899-93963284b5d6" (UID: "2f60591a-fc5a-47d9-9899-93963284b5d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:20:30.609507 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:30.609473 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f60591a-fc5a-47d9-9899-93963284b5d6-kube-api-access-vkbml" pod "2f60591a-fc5a-47d9-9899-93963284b5d6" (UID: "2f60591a-fc5a-47d9-9899-93963284b5d6"). InnerVolumeSpecName "kube-api-access-vkbml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:20:30.708421 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:30.708377 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkbml\" (UniqueName: \"kubernetes.io/projected/2f60591a-fc5a-47d9-9899-93963284b5d6-kube-api-access-vkbml\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:20:30.708421 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:30.708411 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f60591a-fc5a-47d9-9899-93963284b5d6-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:20:30.967663 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:30.967617 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-jjmps"] Nov 03 22:20:30.981409 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:30.981377 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-jjmps"] Nov 03 22:20:31.651503 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:31.651467 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f60591a-fc5a-47d9-9899-93963284b5d6" path="/var/lib/kubelet/pods/2f60591a-fc5a-47d9-9899-93963284b5d6/volumes" Nov 03 22:20:39.327647 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:39.327610 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-r5zp7"] Nov 03 22:20:40.618510 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:20:40.618473 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:20:44.328306 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:44.328255 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-7bb9489676-r5zp7" podUID="40e8212c-49a4-4a50-add6-0ee8a55429b2" containerName="payload-processing" containerID="cri-o://6b3c1e65feb0bc440eaf188b149fafedc9c6c53b344ee8ac4590d2ee481ecdad" gracePeriod=25 Nov 03 22:20:44.577412 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:44.577387 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-r5zp7" Nov 03 22:20:44.614331 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:44.614249 5615 generic.go:356] "Generic (PLEG): container finished" podID="40e8212c-49a4-4a50-add6-0ee8a55429b2" containerID="6b3c1e65feb0bc440eaf188b149fafedc9c6c53b344ee8ac4590d2ee481ecdad" exitCode=0 Nov 03 22:20:44.614331 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:44.614296 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-r5zp7" Nov 03 22:20:44.614525 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:44.614334 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-r5zp7" event={"ID":"40e8212c-49a4-4a50-add6-0ee8a55429b2","Type":"ContainerDied","Data":"6b3c1e65feb0bc440eaf188b149fafedc9c6c53b344ee8ac4590d2ee481ecdad"} Nov 03 22:20:44.614525 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:44.614386 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-r5zp7" event={"ID":"40e8212c-49a4-4a50-add6-0ee8a55429b2","Type":"ContainerDied","Data":"6291d611a4569ac07d7d258b0e7c134c693004e3c48fde7fd314d2c695289ce1"} Nov 03 22:20:44.614525 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:44.614412 5615 scope.go:122] "RemoveContainer" containerID="6b3c1e65feb0bc440eaf188b149fafedc9c6c53b344ee8ac4590d2ee481ecdad" Nov 03 22:20:44.623047 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:44.623027 5615 scope.go:122] "RemoveContainer" containerID="6b3c1e65feb0bc440eaf188b149fafedc9c6c53b344ee8ac4590d2ee481ecdad" Nov 03 22:20:44.623354 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:20:44.623334 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b3c1e65feb0bc440eaf188b149fafedc9c6c53b344ee8ac4590d2ee481ecdad\": container with ID starting with 6b3c1e65feb0bc440eaf188b149fafedc9c6c53b344ee8ac4590d2ee481ecdad not found: ID does not exist" containerID="6b3c1e65feb0bc440eaf188b149fafedc9c6c53b344ee8ac4590d2ee481ecdad" Nov 03 22:20:44.623405 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:44.623364 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b3c1e65feb0bc440eaf188b149fafedc9c6c53b344ee8ac4590d2ee481ecdad"} err="failed to get container status \"6b3c1e65feb0bc440eaf188b149fafedc9c6c53b344ee8ac4590d2ee481ecdad\": rpc error: code = NotFound desc = could not find container \"6b3c1e65feb0bc440eaf188b149fafedc9c6c53b344ee8ac4590d2ee481ecdad\": container with ID starting with 6b3c1e65feb0bc440eaf188b149fafedc9c6c53b344ee8ac4590d2ee481ecdad not found: ID does not exist" Nov 03 22:20:44.753359 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:44.753323 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40e8212c-49a4-4a50-add6-0ee8a55429b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e8212c-49a4-4a50-add6-0ee8a55429b2-plugins-config-volume\") pod \"40e8212c-49a4-4a50-add6-0ee8a55429b2\" (UID: \"40e8212c-49a4-4a50-add6-0ee8a55429b2\") " Nov 03 22:20:44.753572 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:44.753374 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e8212c-49a4-4a50-add6-0ee8a55429b2-kube-api-access-j28sv\" (UniqueName: \"kubernetes.io/projected/40e8212c-49a4-4a50-add6-0ee8a55429b2-kube-api-access-j28sv\") pod \"40e8212c-49a4-4a50-add6-0ee8a55429b2\" (UID: \"40e8212c-49a4-4a50-add6-0ee8a55429b2\") " Nov 03 22:20:44.753756 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:44.753727 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e8212c-49a4-4a50-add6-0ee8a55429b2-plugins-config-volume" pod "40e8212c-49a4-4a50-add6-0ee8a55429b2" (UID: "40e8212c-49a4-4a50-add6-0ee8a55429b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:20:44.755769 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:44.755745 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e8212c-49a4-4a50-add6-0ee8a55429b2-kube-api-access-j28sv" pod "40e8212c-49a4-4a50-add6-0ee8a55429b2" (UID: "40e8212c-49a4-4a50-add6-0ee8a55429b2"). InnerVolumeSpecName "kube-api-access-j28sv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:20:44.855105 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:44.855038 5615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e8212c-49a4-4a50-add6-0ee8a55429b2-plugins-config-volume\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:20:44.855264 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:44.855114 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j28sv\" (UniqueName: \"kubernetes.io/projected/40e8212c-49a4-4a50-add6-0ee8a55429b2-kube-api-access-j28sv\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:20:45.036336 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:45.036302 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-r5zp7"] Nov 03 22:20:45.040099 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:45.040062 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-r5zp7"] Nov 03 22:20:45.650134 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:20:45.650096 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40e8212c-49a4-4a50-add6-0ee8a55429b2" path="/var/lib/kubelet/pods/40e8212c-49a4-4a50-add6-0ee8a55429b2/volumes" Nov 03 22:21:10.618298 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:21:10.618257 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:21:40.618515 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:21:40.618425 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:22:10.620857 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:22:10.620809 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:22:27.796659 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:22:27.796621 5615 scope.go:122] "RemoveContainer" containerID="7cd6221163cc550599b8bc351e4a712be24e7cc5a912ffa1e3a99e9dd869a3da" Nov 03 22:22:40.618600 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:22:40.618556 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:23:10.619836 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:23:10.619754 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:23:40.618143 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:23:40.618103 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:24:10.618110 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:24:10.618043 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:24:40.619442 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:24:40.619353 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:25:10.617921 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:25:10.617873 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:25:40.618472 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:25:40.618426 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:26:10.619102 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:26:10.619001 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:26:15.625099 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:26:15.624205 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:26:15.625099 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:26:15.624313 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-n5nd4" podUID="cb73420b-7000-4ae1-99de-e72bdd6b847b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:26:40.633284 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:26:40.633230 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:27:10.618381 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:27:10.618339 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:27:40.618594 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:27:40.618501 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:28:10.618438 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:28:10.618399 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:28:17.684100 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:17.683902 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7"] Nov 03 22:28:17.684812 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:17.684560 5615 cpu_manager.go:414] "containerMap: removing container" podUID="40e8212c-49a4-4a50-add6-0ee8a55429b2" containerName="payload-processing" Nov 03 22:28:17.684812 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:17.684582 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40e8212c-49a4-4a50-add6-0ee8a55429b2" containerName="payload-processing" Nov 03 22:28:17.684812 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:17.684634 5615 cpu_manager.go:414] "containerMap: removing container" podUID="0469d23e-8be1-4425-946c-9c8f9cff3792" containerName="maas-api" Nov 03 22:28:17.684812 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:17.684643 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0469d23e-8be1-4425-946c-9c8f9cff3792" containerName="maas-api" Nov 03 22:28:17.684812 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:17.684676 5615 cpu_manager.go:414] "containerMap: removing container" podUID="2f60591a-fc5a-47d9-9899-93963284b5d6" containerName="payload-pre-processing" Nov 03 22:28:17.684812 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:17.684684 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f60591a-fc5a-47d9-9899-93963284b5d6" containerName="payload-pre-processing" Nov 03 22:28:17.684812 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:17.684792 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f60591a-fc5a-47d9-9899-93963284b5d6" containerName="payload-pre-processing" Nov 03 22:28:17.684812 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:17.684809 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="40e8212c-49a4-4a50-add6-0ee8a55429b2" containerName="payload-processing" Nov 03 22:28:17.693360 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:17.684827 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="0469d23e-8be1-4425-946c-9c8f9cff3792" containerName="maas-api" Nov 03 22:28:17.693360 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:17.693233 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:17.781038 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:17.780993 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-51b683-openshift-default-dockercfg-rd27d\"" Nov 03 22:28:17.840100 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:17.834874 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7"] Nov 03 22:28:17.902746 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:17.902699 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istio-token\") pod \"e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:18.038098 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:18.032689 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-credential-socket\") pod \"e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:18.038098 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:18.032755 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istio-envoy\") pod \"e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:18.038098 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:18.032831 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpp2d\" (UniqueName: \"kubernetes.io/projected/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-kube-api-access-kpp2d\") pod \"e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:18.038098 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:18.032874 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istio-podinfo\") pod \"e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:18.038098 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:18.032903 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-workload-certs\") pod \"e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:18.038098 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:18.032965 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-workload-socket\") pod \"e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:18.038098 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:18.033063 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istio-data\") pod \"e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:18.038098 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:18.033126 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istiod-ca-cert\") pod \"e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:18.134688 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:18.134643 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istio-envoy\") pod \"e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:18.135012 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:18.134985 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpp2d\" (UniqueName: \"kubernetes.io/projected/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-kube-api-access-kpp2d\") pod \"e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:18.140179 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:18.140146 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istio-podinfo\") pod \"e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:18.140378 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:18.140358 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-workload-certs\") pod \"e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:18.140534 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:18.140519 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-workload-socket\") pod \"e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:18.144097 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:18.140695 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istio-data\") pod \"e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:18.164099 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:18.156223 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istiod-ca-cert\") pod \"e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:18.164477 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:18.144686 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-workload-socket\") pod \"e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:18.164477 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:18.144929 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istio-data\") pod \"e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:18.164477 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:18.141766 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-workload-certs\") pod \"e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:18.164477 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:18.157144 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istiod-ca-cert\") pod \"e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:18.168096 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:18.164786 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istio-token\") pod \"e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:18.175099 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:18.168311 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-credential-socket\") pod \"e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:18.175789 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:18.171493 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-credential-socket\") pod \"e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:18.228106 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:18.209755 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istio-podinfo\") pod \"e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:18.228106 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:18.210000 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istio-envoy\") pod \"e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:18.228106 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:18.215663 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpp2d\" (UniqueName: \"kubernetes.io/projected/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-kube-api-access-kpp2d\") pod \"e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:18.247701 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:18.247626 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istio-token\") pod \"e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:18.445151 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:18.444255 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:19.387112 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:19.366530 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7"] Nov 03 22:28:19.414998 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:19.414948 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:28:19.415289 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:19.415265 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:28:19.415440 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:19.415423 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:28:19.445404 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:19.445324 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" event={"ID":"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e","Type":"ContainerStarted","Data":"5f2e26d85f4471826a65213652f49776280bf3cb8d83818b6ccb96c571aa1b3c"} Nov 03 22:28:20.452083 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:20.452029 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" event={"ID":"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e","Type":"ContainerStarted","Data":"d9aa9acaf59cc9570dc6777bbbf9b15a58bcc80ef57a4aa788cb372c79eabb22"} Nov 03 22:28:20.625704 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:20.625646 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" podStartSLOduration=3.625633997 podStartE2EDuration="3.625633997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:20.625070308 +0000 UTC m=+5995.092849031" watchObservedRunningTime="2026-11-03 22:28:20.625633997 +0000 UTC m=+5995.093412720" Nov 03 22:28:21.445798 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:21.445743 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:21.450981 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:21.450950 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:21.457327 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:21.457296 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:21.458308 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:21.458283 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:30.129948 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:30.129909 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl"] Nov 03 22:28:30.134922 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:30.134891 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl" Nov 03 22:28:30.188720 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:30.188686 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl"] Nov 03 22:28:30.238236 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:30.238194 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-51b683-serving-cert\"" Nov 03 22:28:30.250903 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:30.250853 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m82r5\" (UniqueName: \"kubernetes.io/projected/a941e28b-982a-4437-bb13-55f2d3d33e02-kube-api-access-m82r5\") pod \"maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl\" (UID: \"a941e28b-982a-4437-bb13-55f2d3d33e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl" Nov 03 22:28:30.251142 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:30.250919 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a941e28b-982a-4437-bb13-55f2d3d33e02-metrics-certs\") pod \"maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl\" (UID: \"a941e28b-982a-4437-bb13-55f2d3d33e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl" Nov 03 22:28:30.251142 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:30.251059 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a941e28b-982a-4437-bb13-55f2d3d33e02-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl\" (UID: \"a941e28b-982a-4437-bb13-55f2d3d33e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl" Nov 03 22:28:30.251256 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:30.251169 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a941e28b-982a-4437-bb13-55f2d3d33e02-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl\" (UID: \"a941e28b-982a-4437-bb13-55f2d3d33e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl" Nov 03 22:28:30.352101 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:30.352045 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m82r5\" (UniqueName: \"kubernetes.io/projected/a941e28b-982a-4437-bb13-55f2d3d33e02-kube-api-access-m82r5\") pod \"maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl\" (UID: \"a941e28b-982a-4437-bb13-55f2d3d33e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl" Nov 03 22:28:30.352298 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:30.352132 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a941e28b-982a-4437-bb13-55f2d3d33e02-metrics-certs\") pod \"maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl\" (UID: \"a941e28b-982a-4437-bb13-55f2d3d33e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl" Nov 03 22:28:30.352298 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:30.352192 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a941e28b-982a-4437-bb13-55f2d3d33e02-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl\" (UID: \"a941e28b-982a-4437-bb13-55f2d3d33e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl" Nov 03 22:28:30.352298 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:30.352217 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a941e28b-982a-4437-bb13-55f2d3d33e02-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl\" (UID: \"a941e28b-982a-4437-bb13-55f2d3d33e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl" Nov 03 22:28:30.352434 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:28:30.352347 5615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-serving-cert: secret "maas-api-e2e-worker-single-main-51b683-serving-cert" not found Nov 03 22:28:30.352434 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:28:30.352428 5615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a941e28b-982a-4437-bb13-55f2d3d33e02-maas-api-tls podName:a941e28b-982a-4437-bb13-55f2d3d33e02 nodeName:}" failed. No retries permitted until 2026-11-03 22:28:30.852405521 +0000 UTC m=+6005.320184239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a941e28b-982a-4437-bb13-55f2d3d33e02-maas-api-tls") pod "maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl" (UID: "a941e28b-982a-4437-bb13-55f2d3d33e02") : secret "maas-api-e2e-worker-single-main-51b683-serving-cert" not found Nov 03 22:28:30.352956 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:30.352937 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a941e28b-982a-4437-bb13-55f2d3d33e02-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl\" (UID: \"a941e28b-982a-4437-bb13-55f2d3d33e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl" Nov 03 22:28:30.354657 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:30.354634 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a941e28b-982a-4437-bb13-55f2d3d33e02-metrics-certs\") pod \"maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl\" (UID: \"a941e28b-982a-4437-bb13-55f2d3d33e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl" Nov 03 22:28:30.417271 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:30.417193 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m82r5\" (UniqueName: \"kubernetes.io/projected/a941e28b-982a-4437-bb13-55f2d3d33e02-kube-api-access-m82r5\") pod \"maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl\" (UID: \"a941e28b-982a-4437-bb13-55f2d3d33e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl" Nov 03 22:28:30.856427 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:30.856380 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a941e28b-982a-4437-bb13-55f2d3d33e02-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl\" (UID: \"a941e28b-982a-4437-bb13-55f2d3d33e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl" Nov 03 22:28:30.859013 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:30.858986 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a941e28b-982a-4437-bb13-55f2d3d33e02-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl\" (UID: \"a941e28b-982a-4437-bb13-55f2d3d33e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl" Nov 03 22:28:31.048156 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:31.048119 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl" Nov 03 22:28:31.424054 ip-10-0-129-192 kubenswrapper[5615]: W1103 22:28:31.424012 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda941e28b_982a_4437_bb13_55f2d3d33e02.slice/crio-7f3758ec3c862aecfca32fe3c5ac8d0c6851887fb731e125e5a0901df8612ced WatchSource:0}: Error finding container 7f3758ec3c862aecfca32fe3c5ac8d0c6851887fb731e125e5a0901df8612ced: Status 404 returned error can't find the container with id 7f3758ec3c862aecfca32fe3c5ac8d0c6851887fb731e125e5a0901df8612ced Nov 03 22:28:31.425070 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:31.425045 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl"] Nov 03 22:28:31.425668 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:31.425651 5615 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:28:31.509395 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:31.509352 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl" event={"ID":"a941e28b-982a-4437-bb13-55f2d3d33e02","Type":"ContainerStarted","Data":"7f3758ec3c862aecfca32fe3c5ac8d0c6851887fb731e125e5a0901df8612ced"} Nov 03 22:28:32.515786 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:32.515739 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl" event={"ID":"a941e28b-982a-4437-bb13-55f2d3d33e02","Type":"ContainerStarted","Data":"943c4b8ade3d3527ad93e065dbc7e40f77dac7ece8a64e7c27bad3f034a9b5cc"} Nov 03 22:28:32.516258 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:32.515878 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl" Nov 03 22:28:32.636889 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:32.636830 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl" podStartSLOduration=1.6672791230000001 podStartE2EDuration="2.636814925s" totalImagesPullingTime="969.535802ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:32.636145176 +0000 UTC m=+6007.103923893" watchObservedRunningTime="2026-11-03 22:28:32.636814925 +0000 UTC m=+6007.104593647" Nov 03 22:28:38.527713 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:38.527677 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl" Nov 03 22:28:40.644171 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:28:40.644122 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:28:46.591394 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:46.591357 5615 generic.go:356] "Generic (PLEG): container finished" podID="9f26f6e9-5993-499a-b9a8-fbdd6c801c9e" containerID="d9aa9acaf59cc9570dc6777bbbf9b15a58bcc80ef57a4aa788cb372c79eabb22" exitCode=0 Nov 03 22:28:46.591953 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:46.591436 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" event={"ID":"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e","Type":"ContainerDied","Data":"d9aa9acaf59cc9570dc6777bbbf9b15a58bcc80ef57a4aa788cb372c79eabb22"} Nov 03 22:28:46.591953 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:46.591932 5615 scope.go:122] "RemoveContainer" containerID="d9aa9acaf59cc9570dc6777bbbf9b15a58bcc80ef57a4aa788cb372c79eabb22" Nov 03 22:28:46.593208 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:46.593176 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:28:46.593297 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:46.593249 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:28:46.593297 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:46.593281 5615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:28:47.623550 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:47.623507 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" event={"ID":"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e","Type":"ContainerStarted","Data":"5834cd9a9ee8e2e1905abee2434feba117fc1bf1390e544a39c2c158cafad487"} Nov 03 22:28:48.445016 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:48.444973 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:48.446439 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:48.446410 5615 patch_prober.go:29] interesting pod/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Nov 03 22:28:48.446584 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:48.446476 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" podUID="9f26f6e9-5993-499a-b9a8-fbdd6c801c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Nov 03 22:28:49.445858 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:49.445816 5615 patch_prober.go:29] interesting pod/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Nov 03 22:28:49.446332 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:49.445889 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" podUID="9f26f6e9-5993-499a-b9a8-fbdd6c801c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Nov 03 22:28:50.445298 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:50.445263 5615 patch_prober.go:29] interesting pod/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Nov 03 22:28:50.445486 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:50.445317 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" podUID="9f26f6e9-5993-499a-b9a8-fbdd6c801c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Nov 03 22:28:51.445372 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:51.445334 5615 patch_prober.go:29] interesting pod/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Nov 03 22:28:51.445799 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:51.445392 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" podUID="9f26f6e9-5993-499a-b9a8-fbdd6c801c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Nov 03 22:28:51.457628 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:51.457599 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:52.446097 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:52.446038 5615 patch_prober.go:29] interesting pod/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Nov 03 22:28:52.446616 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:52.446119 5615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" podUID="9f26f6e9-5993-499a-b9a8-fbdd6c801c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Nov 03 22:28:53.448944 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:53.448917 5615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:28:53.449983 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:28:53.449963 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:29:10.618994 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:29:10.618908 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:29:40.618548 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:29:40.618505 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:30:00.744251 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:00.744212 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl"] Nov 03 22:30:00.744830 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:00.744452 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl" podUID="a941e28b-982a-4437-bb13-55f2d3d33e02" containerName="maas-api" containerID="cri-o://943c4b8ade3d3527ad93e065dbc7e40f77dac7ece8a64e7c27bad3f034a9b5cc" gracePeriod=30 Nov 03 22:30:02.001550 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:02.001511 5615 generic.go:356] "Generic (PLEG): container finished" podID="a941e28b-982a-4437-bb13-55f2d3d33e02" containerID="943c4b8ade3d3527ad93e065dbc7e40f77dac7ece8a64e7c27bad3f034a9b5cc" exitCode=0 Nov 03 22:30:02.001937 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:02.001582 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl" event={"ID":"a941e28b-982a-4437-bb13-55f2d3d33e02","Type":"ContainerDied","Data":"943c4b8ade3d3527ad93e065dbc7e40f77dac7ece8a64e7c27bad3f034a9b5cc"} Nov 03 22:30:02.001937 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:02.001635 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl" event={"ID":"a941e28b-982a-4437-bb13-55f2d3d33e02","Type":"ContainerDied","Data":"7f3758ec3c862aecfca32fe3c5ac8d0c6851887fb731e125e5a0901df8612ced"} Nov 03 22:30:02.001937 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:02.001661 5615 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f3758ec3c862aecfca32fe3c5ac8d0c6851887fb731e125e5a0901df8612ced" Nov 03 22:30:02.004534 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:02.004514 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl" Nov 03 22:30:02.046857 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:02.046811 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a941e28b-982a-4437-bb13-55f2d3d33e02-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a941e28b-982a-4437-bb13-55f2d3d33e02-openshift-service-ca\") pod \"a941e28b-982a-4437-bb13-55f2d3d33e02\" (UID: \"a941e28b-982a-4437-bb13-55f2d3d33e02\") " Nov 03 22:30:02.047042 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:02.046902 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a941e28b-982a-4437-bb13-55f2d3d33e02-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a941e28b-982a-4437-bb13-55f2d3d33e02-metrics-certs\") pod \"a941e28b-982a-4437-bb13-55f2d3d33e02\" (UID: \"a941e28b-982a-4437-bb13-55f2d3d33e02\") " Nov 03 22:30:02.047042 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:02.046928 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a941e28b-982a-4437-bb13-55f2d3d33e02-kube-api-access-m82r5\" (UniqueName: \"kubernetes.io/projected/a941e28b-982a-4437-bb13-55f2d3d33e02-kube-api-access-m82r5\") pod \"a941e28b-982a-4437-bb13-55f2d3d33e02\" (UID: \"a941e28b-982a-4437-bb13-55f2d3d33e02\") " Nov 03 22:30:02.047042 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:02.046964 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a941e28b-982a-4437-bb13-55f2d3d33e02-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a941e28b-982a-4437-bb13-55f2d3d33e02-maas-api-tls\") pod \"a941e28b-982a-4437-bb13-55f2d3d33e02\" (UID: \"a941e28b-982a-4437-bb13-55f2d3d33e02\") " Nov 03 22:30:02.047217 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:02.047200 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a941e28b-982a-4437-bb13-55f2d3d33e02-openshift-service-ca" pod "a941e28b-982a-4437-bb13-55f2d3d33e02" (UID: "a941e28b-982a-4437-bb13-55f2d3d33e02"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:02.047407 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:02.047362 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a941e28b-982a-4437-bb13-55f2d3d33e02-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:02.049320 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:02.049288 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a941e28b-982a-4437-bb13-55f2d3d33e02-kube-api-access-m82r5" pod "a941e28b-982a-4437-bb13-55f2d3d33e02" (UID: "a941e28b-982a-4437-bb13-55f2d3d33e02"). InnerVolumeSpecName "kube-api-access-m82r5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:02.049320 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:02.049286 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a941e28b-982a-4437-bb13-55f2d3d33e02-maas-api-tls" pod "a941e28b-982a-4437-bb13-55f2d3d33e02" (UID: "a941e28b-982a-4437-bb13-55f2d3d33e02"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:30:02.049320 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:02.049300 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a941e28b-982a-4437-bb13-55f2d3d33e02-metrics-certs" pod "a941e28b-982a-4437-bb13-55f2d3d33e02" (UID: "a941e28b-982a-4437-bb13-55f2d3d33e02"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:30:02.148343 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:02.148266 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a941e28b-982a-4437-bb13-55f2d3d33e02-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:02.148343 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:02.148296 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m82r5\" (UniqueName: \"kubernetes.io/projected/a941e28b-982a-4437-bb13-55f2d3d33e02-kube-api-access-m82r5\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:02.148343 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:02.148307 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a941e28b-982a-4437-bb13-55f2d3d33e02-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:03.006521 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:03.006488 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl" Nov 03 22:30:03.174275 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:03.174237 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl"] Nov 03 22:30:03.225326 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:03.225291 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-51b683-7875769f7f-cm9wl"] Nov 03 22:30:03.650459 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:03.650421 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a941e28b-982a-4437-bb13-55f2d3d33e02" path="/var/lib/kubelet/pods/a941e28b-982a-4437-bb13-55f2d3d33e02/volumes" Nov 03 22:30:10.617687 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:30:10.617647 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:30:34.746358 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:34.746317 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7"] Nov 03 22:30:34.747031 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:34.746536 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" podUID="9f26f6e9-5993-499a-b9a8-fbdd6c801c9e" containerName="istio-proxy" containerID="cri-o://5834cd9a9ee8e2e1905abee2434feba117fc1bf1390e544a39c2c158cafad487" gracePeriod=30 Nov 03 22:30:38.452272 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:38.452182 5615 patch_prober.go:29] interesting pod/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:30:38.452272 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:38.452251 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" podUID="9f26f6e9-5993-499a-b9a8-fbdd6c801c9e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:30:39.997492 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:39.997462 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:30:40.152846 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.152758 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-kube-api-access-kpp2d\" (UniqueName: \"kubernetes.io/projected/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-kube-api-access-kpp2d\") pod \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " Nov 03 22:30:40.152846 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.152806 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istio-envoy\") pod \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " Nov 03 22:30:40.152846 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.152825 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-workload-socket\") pod \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " Nov 03 22:30:40.153183 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.152866 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-credential-socket\") pod \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " Nov 03 22:30:40.153183 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.152891 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istio-token\" (UniqueName: \"kubernetes.io/projected/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istio-token\") pod \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " Nov 03 22:30:40.153183 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.152932 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istio-podinfo\") pod \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " Nov 03 22:30:40.153183 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.152969 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istio-data\") pod \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " Nov 03 22:30:40.153183 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.152999 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-workload-certs\") pod \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " Nov 03 22:30:40.153183 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.153070 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istiod-ca-cert\") pod \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\" (UID: \"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e\") " Nov 03 22:30:40.153505 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.153302 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-workload-socket" pod "9f26f6e9-5993-499a-b9a8-fbdd6c801c9e" (UID: "9f26f6e9-5993-499a-b9a8-fbdd6c801c9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:30:40.153578 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.153498 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istio-data" pod "9f26f6e9-5993-499a-b9a8-fbdd6c801c9e" (UID: "9f26f6e9-5993-499a-b9a8-fbdd6c801c9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:30:40.153578 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.153295 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-credential-socket" pod "9f26f6e9-5993-499a-b9a8-fbdd6c801c9e" (UID: "9f26f6e9-5993-499a-b9a8-fbdd6c801c9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:30:40.153578 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.153513 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-workload-certs" pod "9f26f6e9-5993-499a-b9a8-fbdd6c801c9e" (UID: "9f26f6e9-5993-499a-b9a8-fbdd6c801c9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:30:40.153756 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.153655 5615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-credential-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:40.153756 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.153673 5615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istio-data\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:40.153756 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.153682 5615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-workload-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:40.153756 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.153691 5615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-workload-socket\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:40.153756 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.153654 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istiod-ca-cert" pod "9f26f6e9-5993-499a-b9a8-fbdd6c801c9e" (UID: "9f26f6e9-5993-499a-b9a8-fbdd6c801c9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:40.155326 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.155301 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istio-envoy" pod "9f26f6e9-5993-499a-b9a8-fbdd6c801c9e" (UID: "9f26f6e9-5993-499a-b9a8-fbdd6c801c9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:30:40.155326 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.155313 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istio-podinfo" pod "9f26f6e9-5993-499a-b9a8-fbdd6c801c9e" (UID: "9f26f6e9-5993-499a-b9a8-fbdd6c801c9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:30:40.155525 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.155510 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-kube-api-access-kpp2d" pod "9f26f6e9-5993-499a-b9a8-fbdd6c801c9e" (UID: "9f26f6e9-5993-499a-b9a8-fbdd6c801c9e"). InnerVolumeSpecName "kube-api-access-kpp2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:40.155687 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.155674 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istio-token" pod "9f26f6e9-5993-499a-b9a8-fbdd6c801c9e" (UID: "9f26f6e9-5993-499a-b9a8-fbdd6c801c9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:40.195818 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.195779 5615 generic.go:356] "Generic (PLEG): container finished" podID="9f26f6e9-5993-499a-b9a8-fbdd6c801c9e" containerID="5834cd9a9ee8e2e1905abee2434feba117fc1bf1390e544a39c2c158cafad487" exitCode=0 Nov 03 22:30:40.196007 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.195824 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" event={"ID":"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e","Type":"ContainerDied","Data":"5834cd9a9ee8e2e1905abee2434feba117fc1bf1390e544a39c2c158cafad487"} Nov 03 22:30:40.196007 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.195836 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" Nov 03 22:30:40.196007 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.195871 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7" event={"ID":"9f26f6e9-5993-499a-b9a8-fbdd6c801c9e","Type":"ContainerDied","Data":"5f2e26d85f4471826a65213652f49776280bf3cb8d83818b6ccb96c571aa1b3c"} Nov 03 22:30:40.196007 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.195895 5615 scope.go:122] "RemoveContainer" containerID="5834cd9a9ee8e2e1905abee2434feba117fc1bf1390e544a39c2c158cafad487" Nov 03 22:30:40.205589 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.205568 5615 scope.go:122] "RemoveContainer" containerID="d9aa9acaf59cc9570dc6777bbbf9b15a58bcc80ef57a4aa788cb372c79eabb22" Nov 03 22:30:40.213977 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.213953 5615 scope.go:122] "RemoveContainer" containerID="5834cd9a9ee8e2e1905abee2434feba117fc1bf1390e544a39c2c158cafad487" Nov 03 22:30:40.214291 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:30:40.214269 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5834cd9a9ee8e2e1905abee2434feba117fc1bf1390e544a39c2c158cafad487\": container with ID starting with 5834cd9a9ee8e2e1905abee2434feba117fc1bf1390e544a39c2c158cafad487 not found: ID does not exist" containerID="5834cd9a9ee8e2e1905abee2434feba117fc1bf1390e544a39c2c158cafad487" Nov 03 22:30:40.214384 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.214305 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5834cd9a9ee8e2e1905abee2434feba117fc1bf1390e544a39c2c158cafad487"} err="failed to get container status \"5834cd9a9ee8e2e1905abee2434feba117fc1bf1390e544a39c2c158cafad487\": rpc error: code = NotFound desc = could not find container \"5834cd9a9ee8e2e1905abee2434feba117fc1bf1390e544a39c2c158cafad487\": container with ID starting with 5834cd9a9ee8e2e1905abee2434feba117fc1bf1390e544a39c2c158cafad487 not found: ID does not exist" Nov 03 22:30:40.214384 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.214332 5615 scope.go:122] "RemoveContainer" containerID="d9aa9acaf59cc9570dc6777bbbf9b15a58bcc80ef57a4aa788cb372c79eabb22" Nov 03 22:30:40.214636 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:30:40.214600 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9aa9acaf59cc9570dc6777bbbf9b15a58bcc80ef57a4aa788cb372c79eabb22\": container with ID starting with d9aa9acaf59cc9570dc6777bbbf9b15a58bcc80ef57a4aa788cb372c79eabb22 not found: ID does not exist" containerID="d9aa9acaf59cc9570dc6777bbbf9b15a58bcc80ef57a4aa788cb372c79eabb22" Nov 03 22:30:40.214676 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.214643 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9aa9acaf59cc9570dc6777bbbf9b15a58bcc80ef57a4aa788cb372c79eabb22"} err="failed to get container status \"d9aa9acaf59cc9570dc6777bbbf9b15a58bcc80ef57a4aa788cb372c79eabb22\": rpc error: code = NotFound desc = could not find container \"d9aa9acaf59cc9570dc6777bbbf9b15a58bcc80ef57a4aa788cb372c79eabb22\": container with ID starting with d9aa9acaf59cc9570dc6777bbbf9b15a58bcc80ef57a4aa788cb372c79eabb22 not found: ID does not exist" Nov 03 22:30:40.255053 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.255017 5615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istiod-ca-cert\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:40.255053 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.255050 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpp2d\" (UniqueName: \"kubernetes.io/projected/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-kube-api-access-kpp2d\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:40.255279 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.255061 5615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istio-envoy\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:40.255279 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.255115 5615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istio-token\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:40.255279 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.255126 5615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e-istio-podinfo\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:40.364752 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.364713 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7"] Nov 03 22:30:40.368998 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:40.368969 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-51b683-openshift-default-554fffdbd9k66x7"] Nov 03 22:30:40.618507 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:30:40.618469 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:30:41.649871 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:41.649840 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f26f6e9-5993-499a-b9a8-fbdd6c801c9e" path="/var/lib/kubelet/pods/9f26f6e9-5993-499a-b9a8-fbdd6c801c9e/volumes" Nov 03 22:30:53.302235 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:53.302191 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:30:53.303284 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:30:53.303146 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" podUID="6b73a9a0-b746-4c20-bfae-f3c670ef04cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:10.618391 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:31:10.618346 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:31:40.618343 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:31:40.618295 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:32:10.618454 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:32:10.618356 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:32:40.619333 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:32:40.619288 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:32:53.301099 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:32:53.300815 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:32:53.301099 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:32:53.300911 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" podUID="6b73a9a0-b746-4c20-bfae-f3c670ef04cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:33:07.470360 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:07.470326 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7c798d54f8-4sbjv"] Nov 03 22:33:07.470905 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:07.470747 5615 cpu_manager.go:414] "containerMap: removing container" podUID="9f26f6e9-5993-499a-b9a8-fbdd6c801c9e" containerName="istio-proxy" Nov 03 22:33:07.470905 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:07.470760 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f26f6e9-5993-499a-b9a8-fbdd6c801c9e" containerName="istio-proxy" Nov 03 22:33:07.470905 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:07.470786 5615 cpu_manager.go:414] "containerMap: removing container" podUID="a941e28b-982a-4437-bb13-55f2d3d33e02" containerName="maas-api" Nov 03 22:33:07.470905 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:07.470794 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a941e28b-982a-4437-bb13-55f2d3d33e02" containerName="maas-api" Nov 03 22:33:07.470905 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:07.470870 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f26f6e9-5993-499a-b9a8-fbdd6c801c9e" containerName="istio-proxy" Nov 03 22:33:07.470905 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:07.470881 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f26f6e9-5993-499a-b9a8-fbdd6c801c9e" containerName="istio-proxy" Nov 03 22:33:07.470905 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:07.470890 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="a941e28b-982a-4437-bb13-55f2d3d33e02" containerName="maas-api" Nov 03 22:33:07.474126 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:07.474096 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c798d54f8-4sbjv" Nov 03 22:33:07.580272 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:07.580239 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c798d54f8-4sbjv"] Nov 03 22:33:07.623245 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:07.623204 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2xgs\" (UniqueName: \"kubernetes.io/projected/44a656ce-295d-4107-be72-fee9c672b5b1-kube-api-access-h2xgs\") pod \"maas-api-7c798d54f8-4sbjv\" (UID: \"44a656ce-295d-4107-be72-fee9c672b5b1\") " pod="odh-ai-gateway-infra/maas-api-7c798d54f8-4sbjv" Nov 03 22:33:07.623431 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:07.623274 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44a656ce-295d-4107-be72-fee9c672b5b1-metrics-certs\") pod \"maas-api-7c798d54f8-4sbjv\" (UID: \"44a656ce-295d-4107-be72-fee9c672b5b1\") " pod="odh-ai-gateway-infra/maas-api-7c798d54f8-4sbjv" Nov 03 22:33:07.623431 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:07.623377 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44a656ce-295d-4107-be72-fee9c672b5b1-maas-api-tls\") pod \"maas-api-7c798d54f8-4sbjv\" (UID: \"44a656ce-295d-4107-be72-fee9c672b5b1\") " pod="odh-ai-gateway-infra/maas-api-7c798d54f8-4sbjv" Nov 03 22:33:07.623504 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:07.623430 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44a656ce-295d-4107-be72-fee9c672b5b1-openshift-service-ca\") pod \"maas-api-7c798d54f8-4sbjv\" (UID: \"44a656ce-295d-4107-be72-fee9c672b5b1\") " pod="odh-ai-gateway-infra/maas-api-7c798d54f8-4sbjv" Nov 03 22:33:07.724566 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:07.724478 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2xgs\" (UniqueName: \"kubernetes.io/projected/44a656ce-295d-4107-be72-fee9c672b5b1-kube-api-access-h2xgs\") pod \"maas-api-7c798d54f8-4sbjv\" (UID: \"44a656ce-295d-4107-be72-fee9c672b5b1\") " pod="odh-ai-gateway-infra/maas-api-7c798d54f8-4sbjv" Nov 03 22:33:07.724566 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:07.724547 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44a656ce-295d-4107-be72-fee9c672b5b1-metrics-certs\") pod \"maas-api-7c798d54f8-4sbjv\" (UID: \"44a656ce-295d-4107-be72-fee9c672b5b1\") " pod="odh-ai-gateway-infra/maas-api-7c798d54f8-4sbjv" Nov 03 22:33:07.724790 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:07.724606 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44a656ce-295d-4107-be72-fee9c672b5b1-maas-api-tls\") pod \"maas-api-7c798d54f8-4sbjv\" (UID: \"44a656ce-295d-4107-be72-fee9c672b5b1\") " pod="odh-ai-gateway-infra/maas-api-7c798d54f8-4sbjv" Nov 03 22:33:07.724790 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:07.724680 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44a656ce-295d-4107-be72-fee9c672b5b1-openshift-service-ca\") pod \"maas-api-7c798d54f8-4sbjv\" (UID: \"44a656ce-295d-4107-be72-fee9c672b5b1\") " pod="odh-ai-gateway-infra/maas-api-7c798d54f8-4sbjv" Nov 03 22:33:07.725419 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:07.725395 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44a656ce-295d-4107-be72-fee9c672b5b1-openshift-service-ca\") pod \"maas-api-7c798d54f8-4sbjv\" (UID: \"44a656ce-295d-4107-be72-fee9c672b5b1\") " pod="odh-ai-gateway-infra/maas-api-7c798d54f8-4sbjv" Nov 03 22:33:07.727109 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:07.727086 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44a656ce-295d-4107-be72-fee9c672b5b1-maas-api-tls\") pod \"maas-api-7c798d54f8-4sbjv\" (UID: \"44a656ce-295d-4107-be72-fee9c672b5b1\") " pod="odh-ai-gateway-infra/maas-api-7c798d54f8-4sbjv" Nov 03 22:33:07.727198 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:07.727159 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44a656ce-295d-4107-be72-fee9c672b5b1-metrics-certs\") pod \"maas-api-7c798d54f8-4sbjv\" (UID: \"44a656ce-295d-4107-be72-fee9c672b5b1\") " pod="odh-ai-gateway-infra/maas-api-7c798d54f8-4sbjv" Nov 03 22:33:07.787132 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:07.787095 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2xgs\" (UniqueName: \"kubernetes.io/projected/44a656ce-295d-4107-be72-fee9c672b5b1-kube-api-access-h2xgs\") pod \"maas-api-7c798d54f8-4sbjv\" (UID: \"44a656ce-295d-4107-be72-fee9c672b5b1\") " pod="odh-ai-gateway-infra/maas-api-7c798d54f8-4sbjv" Nov 03 22:33:07.788890 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:07.788866 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c798d54f8-4sbjv" Nov 03 22:33:08.159493 ip-10-0-129-192 kubenswrapper[5615]: W1103 22:33:08.159455 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a656ce_295d_4107_be72_fee9c672b5b1.slice/crio-1e6e2947d0974eaf58ecc5365ead5168b3900e6f655954728461855b2c450ed6 WatchSource:0}: Error finding container 1e6e2947d0974eaf58ecc5365ead5168b3900e6f655954728461855b2c450ed6: Status 404 returned error can't find the container with id 1e6e2947d0974eaf58ecc5365ead5168b3900e6f655954728461855b2c450ed6 Nov 03 22:33:08.161266 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:08.161238 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c798d54f8-4sbjv"] Nov 03 22:33:09.044212 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:09.044173 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c798d54f8-4sbjv" event={"ID":"44a656ce-295d-4107-be72-fee9c672b5b1","Type":"ContainerStarted","Data":"1e6e2947d0974eaf58ecc5365ead5168b3900e6f655954728461855b2c450ed6"} Nov 03 22:33:10.050310 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:10.050276 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c798d54f8-4sbjv" event={"ID":"44a656ce-295d-4107-be72-fee9c672b5b1","Type":"ContainerStarted","Data":"18853e6e060041c45f5939d0a7916edb55f72153b13f699f9e9a7f0ba6a40a64"} Nov 03 22:33:10.050712 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:10.050405 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7c798d54f8-4sbjv" Nov 03 22:33:10.171416 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:10.171361 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7c798d54f8-4sbjv" podStartSLOduration=2.209040318 podStartE2EDuration="3.171346884s" totalImagesPullingTime="962.306566ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:10.170593132 +0000 UTC m=+6284.638371855" watchObservedRunningTime="2026-11-03 22:33:10.171346884 +0000 UTC m=+6284.639125607" Nov 03 22:33:10.618797 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:33:10.618756 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:33:16.060672 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:16.060643 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7c798d54f8-4sbjv" Nov 03 22:33:16.233293 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:16.233260 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c97fc4d98-hm842"] Nov 03 22:33:16.233525 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:16.233500 5615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-hm842" podUID="789f79fa-622f-445e-9851-994ffd94dc9b" containerName="maas-api" containerID="cri-o://0ac92368d2f79bfdb36788b906be4dd9049de9853817d071fb531eea7aa8cd41" gracePeriod=30 Nov 03 22:33:17.489923 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:17.489897 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-hm842" Nov 03 22:33:17.620859 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:17.620782 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/789f79fa-622f-445e-9851-994ffd94dc9b-kube-api-access-pkfjv\" (UniqueName: \"kubernetes.io/projected/789f79fa-622f-445e-9851-994ffd94dc9b-kube-api-access-pkfjv\") pod \"789f79fa-622f-445e-9851-994ffd94dc9b\" (UID: \"789f79fa-622f-445e-9851-994ffd94dc9b\") " Nov 03 22:33:17.620859 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:17.620838 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/789f79fa-622f-445e-9851-994ffd94dc9b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/789f79fa-622f-445e-9851-994ffd94dc9b-maas-api-tls\") pod \"789f79fa-622f-445e-9851-994ffd94dc9b\" (UID: \"789f79fa-622f-445e-9851-994ffd94dc9b\") " Nov 03 22:33:17.621125 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:17.620864 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/789f79fa-622f-445e-9851-994ffd94dc9b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/789f79fa-622f-445e-9851-994ffd94dc9b-openshift-service-ca\") pod \"789f79fa-622f-445e-9851-994ffd94dc9b\" (UID: \"789f79fa-622f-445e-9851-994ffd94dc9b\") " Nov 03 22:33:17.621125 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:17.621026 5615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/789f79fa-622f-445e-9851-994ffd94dc9b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/789f79fa-622f-445e-9851-994ffd94dc9b-metrics-certs\") pod \"789f79fa-622f-445e-9851-994ffd94dc9b\" (UID: \"789f79fa-622f-445e-9851-994ffd94dc9b\") " Nov 03 22:33:17.621253 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:17.621217 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789f79fa-622f-445e-9851-994ffd94dc9b-openshift-service-ca" pod "789f79fa-622f-445e-9851-994ffd94dc9b" (UID: "789f79fa-622f-445e-9851-994ffd94dc9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:33:17.621513 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:17.621489 5615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/789f79fa-622f-445e-9851-994ffd94dc9b-openshift-service-ca\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:17.623162 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:17.623133 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/789f79fa-622f-445e-9851-994ffd94dc9b-metrics-certs" pod "789f79fa-622f-445e-9851-994ffd94dc9b" (UID: "789f79fa-622f-445e-9851-994ffd94dc9b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:33:17.623264 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:17.623234 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/789f79fa-622f-445e-9851-994ffd94dc9b-maas-api-tls" pod "789f79fa-622f-445e-9851-994ffd94dc9b" (UID: "789f79fa-622f-445e-9851-994ffd94dc9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:33:17.623416 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:17.623400 5615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789f79fa-622f-445e-9851-994ffd94dc9b-kube-api-access-pkfjv" pod "789f79fa-622f-445e-9851-994ffd94dc9b" (UID: "789f79fa-622f-445e-9851-994ffd94dc9b"). InnerVolumeSpecName "kube-api-access-pkfjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:33:17.722565 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:17.722515 5615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkfjv\" (UniqueName: \"kubernetes.io/projected/789f79fa-622f-445e-9851-994ffd94dc9b-kube-api-access-pkfjv\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:17.722565 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:17.722562 5615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/789f79fa-622f-445e-9851-994ffd94dc9b-maas-api-tls\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:17.722565 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:17.722577 5615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/789f79fa-622f-445e-9851-994ffd94dc9b-metrics-certs\") on node \"ip-10-0-129-192.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:18.093024 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:18.092989 5615 generic.go:356] "Generic (PLEG): container finished" podID="789f79fa-622f-445e-9851-994ffd94dc9b" containerID="0ac92368d2f79bfdb36788b906be4dd9049de9853817d071fb531eea7aa8cd41" exitCode=0 Nov 03 22:33:18.093256 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:18.093061 5615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-hm842" Nov 03 22:33:18.093256 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:18.093101 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-hm842" event={"ID":"789f79fa-622f-445e-9851-994ffd94dc9b","Type":"ContainerDied","Data":"0ac92368d2f79bfdb36788b906be4dd9049de9853817d071fb531eea7aa8cd41"} Nov 03 22:33:18.093256 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:18.093183 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-hm842" event={"ID":"789f79fa-622f-445e-9851-994ffd94dc9b","Type":"ContainerDied","Data":"51f4d00df55beef5c8dbeafe13a69fb760d1e52937ad4358dfadb2f4c4b69d07"} Nov 03 22:33:18.093256 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:18.093208 5615 scope.go:122] "RemoveContainer" containerID="0ac92368d2f79bfdb36788b906be4dd9049de9853817d071fb531eea7aa8cd41" Nov 03 22:33:18.102568 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:18.102545 5615 scope.go:122] "RemoveContainer" containerID="0ac92368d2f79bfdb36788b906be4dd9049de9853817d071fb531eea7aa8cd41" Nov 03 22:33:18.102850 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:33:18.102830 5615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ac92368d2f79bfdb36788b906be4dd9049de9853817d071fb531eea7aa8cd41\": container with ID starting with 0ac92368d2f79bfdb36788b906be4dd9049de9853817d071fb531eea7aa8cd41 not found: ID does not exist" containerID="0ac92368d2f79bfdb36788b906be4dd9049de9853817d071fb531eea7aa8cd41" Nov 03 22:33:18.102929 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:18.102859 5615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ac92368d2f79bfdb36788b906be4dd9049de9853817d071fb531eea7aa8cd41"} err="failed to get container status \"0ac92368d2f79bfdb36788b906be4dd9049de9853817d071fb531eea7aa8cd41\": rpc error: code = NotFound desc = could not find container \"0ac92368d2f79bfdb36788b906be4dd9049de9853817d071fb531eea7aa8cd41\": container with ID starting with 0ac92368d2f79bfdb36788b906be4dd9049de9853817d071fb531eea7aa8cd41 not found: ID does not exist" Nov 03 22:33:18.260164 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:18.260129 5615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c97fc4d98-hm842"] Nov 03 22:33:18.311180 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:18.311140 5615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c97fc4d98-hm842"] Nov 03 22:33:19.649876 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:19.649846 5615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="789f79fa-622f-445e-9851-994ffd94dc9b" path="/var/lib/kubelet/pods/789f79fa-622f-445e-9851-994ffd94dc9b/volumes" Nov 03 22:33:23.296339 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:23.296283 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:33:23.297056 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:33:23.297011 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" podUID="6b73a9a0-b746-4c20-bfae-f3c670ef04cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:33:40.618451 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:33:40.618409 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:34:10.618365 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:34:10.618319 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:34:38.294273 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:34:38.294230 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:34:38.294951 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:34:38.294910 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" podUID="6b73a9a0-b746-4c20-bfae-f3c670ef04cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:34:40.617852 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:34:40.617811 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:35:10.618488 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:35:10.618402 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:35:27.953687 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:35:27.953650 5615 scope.go:122] "RemoveContainer" containerID="943c4b8ade3d3527ad93e065dbc7e40f77dac7ece8a64e7c27bad3f034a9b5cc" Nov 03 22:35:40.618036 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:35:40.617997 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:36:10.618428 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:36:10.618387 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:36:23.288099 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:36:23.287179 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:36:23.288099 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:36:23.287270 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" podUID="6b73a9a0-b746-4c20-bfae-f3c670ef04cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:36:40.619051 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:36:40.618967 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:36:53.295390 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:36:53.295287 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:36:53.296147 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:36:53.295423 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q9mnn" podUID="6b73a9a0-b746-4c20-bfae-f3c670ef04cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:37:10.618713 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:37:10.618672 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:37:40.618514 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:37:40.618472 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:38:10.617864 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:38:10.617784 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:38:40.680099 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:38:40.675205 5615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:38:40.680099 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:38:40.675297 5615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5q7qp" podUID="e0aabe10-5135-4ca5-91a2-226a31408d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:38:40.703931 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:38:40.703818 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:39:10.787391 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:39:10.787292 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:39:36.367974 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:39:36.367892 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-b59f75cb9-vv6wb_20ef8f90-2fb3-4956-9aa6-cd9578f7c17b/manager/0.log" Nov 03 22:39:36.750143 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:39:36.750107 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-x87fj_79c0375b-54c7-4629-83da-f41b5b7e8da9/manager/0.log" Nov 03 22:39:36.992934 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:39:36.992897 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-9qt8v_42cba095-3e85-4cd0-a6fa-255c8086733b/manager/0.log" Nov 03 22:39:38.283863 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:39:38.283834 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-lksbc_db528ef3-1b85-4e45-be0c-602a4eeb95c6/server/0.log" Nov 03 22:39:40.262420 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:39:40.262381 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7c798d54f8-4sbjv_44a656ce-295d-4107-be72-fee9c672b5b1/maas-api/0.log" Nov 03 22:39:40.618483 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:39:40.618395 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:39:49.499812 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:39:49.499771 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-lqd6b_fbc7c3d8-118e-4425-9d20-1547e5a6078c/discovery/0.log" Nov 03 22:39:50.079506 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:39:50.079471 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d69f97dbf-d25vp_0fcbdbb2-d97d-4e8a-8f30-497ca0d41624/kube-auth-proxy/0.log" Nov 03 22:39:50.322216 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:39:50.322189 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-5q7qp_e0aabe10-5135-4ca5-91a2-226a31408d6b/istio-proxy/0.log" Nov 03 22:39:51.740312 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:39:51.740279 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-n5nd4_cb73420b-7000-4ae1-99de-e72bdd6b847b/istio-proxy/0.log" Nov 03 22:39:51.978890 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:39:51.978836 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-q9mnn_6b73a9a0-b746-4c20-bfae-f3c670ef04cf/istio-proxy/0.log" Nov 03 22:39:52.226395 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:39:52.226316 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-r666f_87f7ca28-e370-4e4c-85f3-818e70156336/istio-proxy/0.log" Nov 03 22:39:54.876327 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:39:54.876298 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-wnxz7_42d88418-b5da-43a8-9ff4-edf3bb7a3c56/main/0.log" Nov 03 22:39:55.440581 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:39:55.440545 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-hnr66_dccff396-4195-41f2-ada5-110b2c0ab303/main/0.log" Nov 03 22:39:56.004771 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:39:56.004731 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vhn7f_46bf6611-811b-4e94-85c2-f6ad5871f3e8/main/0.log" Nov 03 22:39:56.550785 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:39:56.550755 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-j8r2v_66f8b152-0f96-4ab7-b6de-a7beb77d81c6/main/0.log" Nov 03 22:40:06.890259 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:06.890219 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cd72m/must-gather-psbpm"] Nov 03 22:40:06.890671 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:06.890637 5615 cpu_manager.go:414] "containerMap: removing container" podUID="789f79fa-622f-445e-9851-994ffd94dc9b" containerName="maas-api" Nov 03 22:40:06.890671 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:06.890648 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789f79fa-622f-445e-9851-994ffd94dc9b" containerName="maas-api" Nov 03 22:40:06.890671 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:06.890666 5615 cpu_manager.go:414] "containerMap: removing container" podUID="9f26f6e9-5993-499a-b9a8-fbdd6c801c9e" containerName="istio-proxy" Nov 03 22:40:06.890671 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:06.890671 5615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f26f6e9-5993-499a-b9a8-fbdd6c801c9e" containerName="istio-proxy" Nov 03 22:40:06.890812 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:06.890745 5615 memory_manager.go:365] "RemoveStaleState removing state" podUID="789f79fa-622f-445e-9851-994ffd94dc9b" containerName="maas-api" Nov 03 22:40:06.894198 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:06.894178 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cd72m/must-gather-psbpm" Nov 03 22:40:06.947292 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:06.947250 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cd72m/must-gather-psbpm"] Nov 03 22:40:06.961399 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:06.961361 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwgj7\" (UniqueName: \"kubernetes.io/projected/db9b4a32-05aa-49ad-b438-dd7b9b2628f1-kube-api-access-gwgj7\") pod \"must-gather-psbpm\" (UID: \"db9b4a32-05aa-49ad-b438-dd7b9b2628f1\") " pod="openshift-must-gather-cd72m/must-gather-psbpm" Nov 03 22:40:06.961579 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:06.961411 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/db9b4a32-05aa-49ad-b438-dd7b9b2628f1-must-gather-output\") pod \"must-gather-psbpm\" (UID: \"db9b4a32-05aa-49ad-b438-dd7b9b2628f1\") " pod="openshift-must-gather-cd72m/must-gather-psbpm" Nov 03 22:40:06.996704 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:06.996671 5615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cd72m\"/\"default-dockercfg-77mtt\"" Nov 03 22:40:06.998479 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:06.998456 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cd72m\"/\"openshift-service-ca.crt\"" Nov 03 22:40:06.998586 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:06.998479 5615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cd72m\"/\"kube-root-ca.crt\"" Nov 03 22:40:07.062969 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:07.062924 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwgj7\" (UniqueName: \"kubernetes.io/projected/db9b4a32-05aa-49ad-b438-dd7b9b2628f1-kube-api-access-gwgj7\") pod \"must-gather-psbpm\" (UID: \"db9b4a32-05aa-49ad-b438-dd7b9b2628f1\") " pod="openshift-must-gather-cd72m/must-gather-psbpm" Nov 03 22:40:07.062969 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:07.062971 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/db9b4a32-05aa-49ad-b438-dd7b9b2628f1-must-gather-output\") pod \"must-gather-psbpm\" (UID: \"db9b4a32-05aa-49ad-b438-dd7b9b2628f1\") " pod="openshift-must-gather-cd72m/must-gather-psbpm" Nov 03 22:40:07.063406 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:07.063382 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/db9b4a32-05aa-49ad-b438-dd7b9b2628f1-must-gather-output\") pod \"must-gather-psbpm\" (UID: \"db9b4a32-05aa-49ad-b438-dd7b9b2628f1\") " pod="openshift-must-gather-cd72m/must-gather-psbpm" Nov 03 22:40:07.125055 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:07.125021 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwgj7\" (UniqueName: \"kubernetes.io/projected/db9b4a32-05aa-49ad-b438-dd7b9b2628f1-kube-api-access-gwgj7\") pod \"must-gather-psbpm\" (UID: \"db9b4a32-05aa-49ad-b438-dd7b9b2628f1\") " pod="openshift-must-gather-cd72m/must-gather-psbpm" Nov 03 22:40:07.209275 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:07.209173 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cd72m/must-gather-psbpm" Nov 03 22:40:07.578668 ip-10-0-129-192 kubenswrapper[5615]: W1103 22:40:07.578637 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb9b4a32_05aa_49ad_b438_dd7b9b2628f1.slice/crio-3e3d6fce6f6f40a35977c43af973682466938082ea49d928e26dd3130f4ac93b WatchSource:0}: Error finding container 3e3d6fce6f6f40a35977c43af973682466938082ea49d928e26dd3130f4ac93b: Status 404 returned error can't find the container with id 3e3d6fce6f6f40a35977c43af973682466938082ea49d928e26dd3130f4ac93b Nov 03 22:40:07.580650 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:07.580633 5615 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:40:07.580706 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:07.580685 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cd72m/must-gather-psbpm"] Nov 03 22:40:07.650262 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:07.650223 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cd72m/must-gather-psbpm" event={"ID":"db9b4a32-05aa-49ad-b438-dd7b9b2628f1","Type":"ContainerStarted","Data":"3e3d6fce6f6f40a35977c43af973682466938082ea49d928e26dd3130f4ac93b"} Nov 03 22:40:09.657030 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:09.656980 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cd72m/must-gather-psbpm" event={"ID":"db9b4a32-05aa-49ad-b438-dd7b9b2628f1","Type":"ContainerStarted","Data":"f2ab36db34768ffabff8e39e82d9a5d804993684c03e715588573a1c4402efef"} Nov 03 22:40:09.657470 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:09.657038 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cd72m/must-gather-psbpm" event={"ID":"db9b4a32-05aa-49ad-b438-dd7b9b2628f1","Type":"ContainerStarted","Data":"1bceb8c8a23f138f1b3024fa879cdfc539212e707a65cd9edb7cd326d903a951"} Nov 03 22:40:09.776544 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:09.776470 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-cd72m/must-gather-psbpm" podStartSLOduration=2.459838797 podStartE2EDuration="3.776449135s" totalImagesPullingTime="1.316610338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:40:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:09.774472797 +0000 UTC m=+6704.242251522" watchObservedRunningTime="2026-11-03 22:40:09.776449135 +0000 UTC m=+6704.244227858" Nov 03 22:40:10.618241 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:40:10.618205 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:40:16.775093 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:16.774966 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mqmds_712fcecb-4757-4dd0-8f96-084b1e6ee903/global-pull-secret-syncer/0.log" Nov 03 22:40:18.532205 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:18.532171 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-b59f75cb9-vv6wb_20ef8f90-2fb3-4956-9aa6-cd9578f7c17b/manager/0.log" Nov 03 22:40:18.664470 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:18.664437 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-x87fj_79c0375b-54c7-4629-83da-f41b5b7e8da9/manager/0.log" Nov 03 22:40:19.089197 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:19.088940 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-9qt8v_42cba095-3e85-4cd0-a6fa-255c8086733b/manager/0.log" Nov 03 22:40:19.895193 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:19.893377 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ksg72_cba96b6a-ccc8-4c1c-b371-1bb5ece29afc/konnectivity-agent/0.log" Nov 03 22:40:20.234878 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:20.234836 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-lksbc_db528ef3-1b85-4e45-be0c-602a4eeb95c6/server/0.log" Nov 03 22:40:20.672010 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:20.671929 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-192.us-west-2.compute.internal_448c85734c0da9850a3328446f82dce9/haproxy/0.log" Nov 03 22:40:24.506992 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:24.506949 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-b59f75cb9-vv6wb_20ef8f90-2fb3-4956-9aa6-cd9578f7c17b/manager/0.log" Nov 03 22:40:24.671787 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:24.671747 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-x87fj_79c0375b-54c7-4629-83da-f41b5b7e8da9/manager/0.log" Nov 03 22:40:24.987019 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:24.986988 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-9qt8v_42cba095-3e85-4cd0-a6fa-255c8086733b/manager/0.log" Nov 03 22:40:26.418145 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:26.418113 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-lksbc_db528ef3-1b85-4e45-be0c-602a4eeb95c6/server/0.log" Nov 03 22:40:26.750379 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:26.750341 5615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cd72m/perf-node-gather-daemonset-5lfg5"] Nov 03 22:40:26.757930 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:26.757900 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-5lfg5" Nov 03 22:40:26.810452 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:26.809824 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cd72m/perf-node-gather-daemonset-5lfg5"] Nov 03 22:40:26.898513 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:26.898467 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvdwd\" (UniqueName: \"kubernetes.io/projected/7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d-kube-api-access-mvdwd\") pod \"perf-node-gather-daemonset-5lfg5\" (UID: \"7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-5lfg5" Nov 03 22:40:26.898704 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:26.898605 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d-proc\") pod \"perf-node-gather-daemonset-5lfg5\" (UID: \"7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-5lfg5" Nov 03 22:40:26.898826 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:26.898740 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d-lib-modules\") pod \"perf-node-gather-daemonset-5lfg5\" (UID: \"7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-5lfg5" Nov 03 22:40:26.898939 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:26.898921 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d-sys\") pod \"perf-node-gather-daemonset-5lfg5\" (UID: \"7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-5lfg5" Nov 03 22:40:26.899054 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:26.898953 5615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d-podres\") pod \"perf-node-gather-daemonset-5lfg5\" (UID: \"7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-5lfg5" Nov 03 22:40:27.000706 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:27.000552 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d-lib-modules\") pod \"perf-node-gather-daemonset-5lfg5\" (UID: \"7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-5lfg5" Nov 03 22:40:27.000941 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:27.000703 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d-lib-modules\") pod \"perf-node-gather-daemonset-5lfg5\" (UID: \"7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-5lfg5" Nov 03 22:40:27.000941 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:27.000728 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d-sys\") pod \"perf-node-gather-daemonset-5lfg5\" (UID: \"7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-5lfg5" Nov 03 22:40:27.000941 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:27.000765 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d-podres\") pod \"perf-node-gather-daemonset-5lfg5\" (UID: \"7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-5lfg5" Nov 03 22:40:27.000941 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:27.000770 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d-sys\") pod \"perf-node-gather-daemonset-5lfg5\" (UID: \"7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-5lfg5" Nov 03 22:40:27.000941 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:27.000823 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvdwd\" (UniqueName: \"kubernetes.io/projected/7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d-kube-api-access-mvdwd\") pod \"perf-node-gather-daemonset-5lfg5\" (UID: \"7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-5lfg5" Nov 03 22:40:27.000941 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:27.000850 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d-podres\") pod \"perf-node-gather-daemonset-5lfg5\" (UID: \"7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-5lfg5" Nov 03 22:40:27.000941 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:27.000890 5615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d-proc\") pod \"perf-node-gather-daemonset-5lfg5\" (UID: \"7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-5lfg5" Nov 03 22:40:27.001398 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:27.001006 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d-proc\") pod \"perf-node-gather-daemonset-5lfg5\" (UID: \"7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-5lfg5" Nov 03 22:40:27.065713 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:27.065676 5615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvdwd\" (UniqueName: \"kubernetes.io/projected/7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d-kube-api-access-mvdwd\") pod \"perf-node-gather-daemonset-5lfg5\" (UID: \"7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-5lfg5" Nov 03 22:40:27.077707 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:27.077669 5615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-5lfg5" Nov 03 22:40:27.472095 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:27.472001 5615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cd72m/perf-node-gather-daemonset-5lfg5"] Nov 03 22:40:27.480260 ip-10-0-129-192 kubenswrapper[5615]: W1103 22:40:27.480208 5615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7c55d4db_52a6_45fa_89b4_f14b5c5ddb9d.slice/crio-7c9b07de790bbdb7aa1d0c80713ebcaf78dd6876ef307b9f6dea5bd5b5220ca0 WatchSource:0}: Error finding container 7c9b07de790bbdb7aa1d0c80713ebcaf78dd6876ef307b9f6dea5bd5b5220ca0: Status 404 returned error can't find the container with id 7c9b07de790bbdb7aa1d0c80713ebcaf78dd6876ef307b9f6dea5bd5b5220ca0 Nov 03 22:40:27.815748 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:27.815714 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-5lfg5" event={"ID":"7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d","Type":"ContainerStarted","Data":"bf6eb7d8ffc29a999d1f237ae900db04690ec5f7aac5bcae6f985e5a0457cf37"} Nov 03 22:40:27.815923 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:27.815767 5615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-5lfg5" event={"ID":"7c55d4db-52a6-45fa-89b4-f14b5c5ddb9d","Type":"ContainerStarted","Data":"7c9b07de790bbdb7aa1d0c80713ebcaf78dd6876ef307b9f6dea5bd5b5220ca0"} Nov 03 22:40:27.815923 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:27.815801 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-5lfg5" Nov 03 22:40:27.933770 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:27.933716 5615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-5lfg5" podStartSLOduration=1.933702631 podStartE2EDuration="1.933702631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:27.932874622 +0000 UTC m=+6722.400653344" watchObservedRunningTime="2026-11-03 22:40:27.933702631 +0000 UTC m=+6722.401481354" Nov 03 22:40:31.010905 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:31.010867 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-b59f75cb9-vv6wb_20ef8f90-2fb3-4956-9aa6-cd9578f7c17b/manager/0.log" Nov 03 22:40:31.179442 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:31.179401 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-x87fj_79c0375b-54c7-4629-83da-f41b5b7e8da9/manager/0.log" Nov 03 22:40:31.240352 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:31.240318 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1ab03d50-196f-497f-bd8c-381588ef9237/alertmanager/0.log" Nov 03 22:40:31.494826 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:31.494787 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-9qt8v_42cba095-3e85-4cd0-a6fa-255c8086733b/manager/0.log" Nov 03 22:40:31.673384 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:31.673359 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1ab03d50-196f-497f-bd8c-381588ef9237/config-reloader/0.log" Nov 03 22:40:32.108470 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:32.108425 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1ab03d50-196f-497f-bd8c-381588ef9237/kube-rbac-proxy-web/0.log" Nov 03 22:40:32.540239 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:32.540203 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1ab03d50-196f-497f-bd8c-381588ef9237/kube-rbac-proxy/0.log" Nov 03 22:40:32.778594 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:32.778568 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-lksbc_db528ef3-1b85-4e45-be0c-602a4eeb95c6/server/0.log" Nov 03 22:40:32.971596 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:32.971510 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1ab03d50-196f-497f-bd8c-381588ef9237/kube-rbac-proxy-metric/0.log" Nov 03 22:40:33.404885 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:33.404853 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1ab03d50-196f-497f-bd8c-381588ef9237/prom-label-proxy/0.log" Nov 03 22:40:33.834442 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:33.834413 5615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-5lfg5" Nov 03 22:40:33.838934 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:33.838890 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1ab03d50-196f-497f-bd8c-381588ef9237/init-config-reloader/0.log" Nov 03 22:40:34.728093 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:34.728040 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-z8p96_c14bf8f0-1335-4ee0-b49d-5d1f2836de1c/kube-state-metrics/0.log" Nov 03 22:40:35.150752 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:35.150720 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-z8p96_c14bf8f0-1335-4ee0-b49d-5d1f2836de1c/kube-rbac-proxy-main/0.log" Nov 03 22:40:35.588964 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:35.588928 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-z8p96_c14bf8f0-1335-4ee0-b49d-5d1f2836de1c/kube-rbac-proxy-self/0.log" Nov 03 22:40:36.023555 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:36.023524 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-b9b549fcc-4cxct_ebf82283-d275-4640-bc44-5a461e4f9f9d/metrics-server/0.log" Nov 03 22:40:36.457420 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:36.457336 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-n8k2s_bee40e70-1cd3-4b89-8808-3abf1dcfbee9/monitoring-plugin/0.log" Nov 03 22:40:38.220057 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:38.220024 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ffhc8_3a695777-f42a-4cb6-8a31-c07c8daf321a/node-exporter/0.log" Nov 03 22:40:38.652496 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:38.652458 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ffhc8_3a695777-f42a-4cb6-8a31-c07c8daf321a/kube-rbac-proxy/0.log" Nov 03 22:40:39.086562 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:39.086520 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ffhc8_3a695777-f42a-4cb6-8a31-c07c8daf321a/init-textfile/0.log" Nov 03 22:40:40.624629 ip-10-0-129-192 kubenswrapper[5615]: E1103 22:40:40.624590 5615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tms4q" containerName="extractor" Nov 03 22:40:40.832628 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:40.832361 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-zbcg6_c77a4d5b-a0d3-468b-aa66-7d607794aba1/kube-rbac-proxy-main/0.log" Nov 03 22:40:41.264507 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:41.264469 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-zbcg6_c77a4d5b-a0d3-468b-aa66-7d607794aba1/kube-rbac-proxy-self/0.log" Nov 03 22:40:41.576773 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:41.576673 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-lqd6b_fbc7c3d8-118e-4425-9d20-1547e5a6078c/discovery/0.log" Nov 03 22:40:41.697714 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:41.697680 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-zbcg6_c77a4d5b-a0d3-468b-aa66-7d607794aba1/openshift-state-metrics/0.log" Nov 03 22:40:41.895462 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:41.895381 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d69f97dbf-d25vp_0fcbdbb2-d97d-4e8a-8f30-497ca0d41624/kube-auth-proxy/0.log" Nov 03 22:40:42.113888 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:42.113853 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-5q7qp_e0aabe10-5135-4ca5-91a2-226a31408d6b/istio-proxy/0.log" Nov 03 22:40:43.200696 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:43.200666 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-n5nd4_cb73420b-7000-4ae1-99de-e72bdd6b847b/istio-proxy/0.log" Nov 03 22:40:43.414629 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:43.414597 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-q9mnn_6b73a9a0-b746-4c20-bfae-f3c670ef04cf/istio-proxy/0.log" Nov 03 22:40:43.627206 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:43.627160 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-r666f_87f7ca28-e370-4e4c-85f3-818e70156336/istio-proxy/0.log" Nov 03 22:40:46.497426 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:46.497388 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d484bf9b5-d4v5f_0327c758-d1f5-481d-9cf2-96e4b5f27872/telemeter-client/0.log" Nov 03 22:40:46.928479 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:46.928399 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d484bf9b5-d4v5f_0327c758-d1f5-481d-9cf2-96e4b5f27872/reload/0.log" Nov 03 22:40:47.399150 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:47.399115 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d484bf9b5-d4v5f_0327c758-d1f5-481d-9cf2-96e4b5f27872/kube-rbac-proxy/0.log" Nov 03 22:40:47.869241 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:47.869208 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d578896-ftx7j_15a5eea7-15b5-45a6-838b-35914cb59be1/thanos-query/0.log" Nov 03 22:40:48.306270 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:48.306243 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d578896-ftx7j_15a5eea7-15b5-45a6-838b-35914cb59be1/kube-rbac-proxy-web/0.log" Nov 03 22:40:48.737610 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:48.737572 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d578896-ftx7j_15a5eea7-15b5-45a6-838b-35914cb59be1/kube-rbac-proxy/0.log" Nov 03 22:40:49.174387 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:49.174302 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d578896-ftx7j_15a5eea7-15b5-45a6-838b-35914cb59be1/prom-label-proxy/0.log" Nov 03 22:40:49.606297 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:49.606250 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d578896-ftx7j_15a5eea7-15b5-45a6-838b-35914cb59be1/kube-rbac-proxy-rules/0.log" Nov 03 22:40:50.111633 ip-10-0-129-192 kubenswrapper[5615]: I1103 22:40:50.111603 5615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d578896-ftx7j_15a5eea7-15b5-45a6-838b-35914cb59be1/kube-rbac-proxy-metrics/0.log"